Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240502 14:13:28.977495 6931 test_util.cc:268] Using random seed: 890231858
I20240502 14:13:28.987759 6931 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:13:28.987833 6931 ts_itest-base.cc:113] --------------
I20240502 14:13:28.987890 6931 ts_itest-base.cc:114] 5 tablet servers
I20240502 14:13:28.987953 6931 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:13:28.988006 6931 ts_itest-base.cc:116] --------------
2024-05-02T14:13:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:13:29Z Disabled control of system clock
I20240502 14:13:29.011412 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.196.254:35493
--webserver_interface=127.6.196.254
--webserver_port=0
--builtin_ntp_servers=127.6.196.212:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.196.254:35493
--enable_range_replica_placement=false with env {}
W20240502 14:13:29.304381 6943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:29.304636 6943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:29.304721 6943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:29.311475 6943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:13:29.311579 6943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:29.311623 6943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:13:29.311688 6943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:13:29.319749 6943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.196.254:35493
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.196.254:35493
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.196.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:29.320966 6943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:29.322613 6943 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:29.331956 6948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:29.332332 6949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:29.332523 6951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:29.333086 6943 server_base.cc:1034] running on GCE node
I20240502 14:13:29.333812 6943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:29.334895 6943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:29.336076 6943 hybrid_clock.cc:648] HybridClock initialized: now 1714659209336023 us; error 37 us; skew 500 ppm
I20240502 14:13:29.336328 6943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:29.338249 6943 webserver.cc:460] Webserver started at http://127.6.196.254:34967/ using document root <none> and password file <none>
I20240502 14:13:29.338878 6943 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:29.339000 6943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:29.339282 6943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:29.341794 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f8c347e1af7a4b6383b4a4e7b6b62ce0"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-kjf7"
I20240502 14:13:29.342458 6943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f8c347e1af7a4b6383b4a4e7b6b62ce0"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-kjf7"
I20240502 14:13:29.346614 6943 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:13:29.349470 6957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:29.350484 6943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:13:29.350623 6943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f8c347e1af7a4b6383b4a4e7b6b62ce0"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-kjf7"
I20240502 14:13:29.350800 6943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:29.370070 6943 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:29.370661 6943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:29.370945 6943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:29.386940 6943 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.254:35493
I20240502 14:13:29.386970 7009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.254:35493 every 8 connection(s)
I20240502 14:13:29.388079 6943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:13:29.391422 7010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:29.395013 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 6943
I20240502 14:13:29.395228 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:13:29.399364 7010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0: Bootstrap starting.
I20240502 14:13:29.402446 7010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:29.403292 7010 log.cc:824] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:29.405470 7010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0: No bootstrap required, opened a new log
I20240502 14:13:29.413491 7010 raft_consensus.cc:348] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c347e1af7a4b6383b4a4e7b6b62ce0" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 35493 } }
I20240502 14:13:29.413755 7010 raft_consensus.cc:374] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:29.413841 7010 raft_consensus.cc:729] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8c347e1af7a4b6383b4a4e7b6b62ce0, State: Initialized, Role: FOLLOWER
I20240502 14:13:29.414314 7010 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c347e1af7a4b6383b4a4e7b6b62ce0" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 35493 } }
I20240502 14:13:29.414520 7010 raft_consensus.cc:388] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:29.414645 7010 raft_consensus.cc:482] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:29.414801 7010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:29.417330 7010 raft_consensus.cc:504] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c347e1af7a4b6383b4a4e7b6b62ce0" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 35493 } }
I20240502 14:13:29.417840 7010 leader_election.cc:302] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8c347e1af7a4b6383b4a4e7b6b62ce0; no voters:
I20240502 14:13:29.418426 7010 leader_election.cc:288] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:29.418576 7015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:29.420230 7010 sys_catalog.cc:564] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:13:29.421269 7015 raft_consensus.cc:686] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [term 1 LEADER]: Becoming Leader. State: Replica: f8c347e1af7a4b6383b4a4e7b6b62ce0, State: Running, Role: LEADER
I20240502 14:13:29.421891 7015 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c347e1af7a4b6383b4a4e7b6b62ce0" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 35493 } }
I20240502 14:13:29.424477 7017 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8c347e1af7a4b6383b4a4e7b6b62ce0. Latest consensus state: current_term: 1 leader_uuid: "f8c347e1af7a4b6383b4a4e7b6b62ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c347e1af7a4b6383b4a4e7b6b62ce0" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 35493 } } }
I20240502 14:13:29.424612 7016 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8c347e1af7a4b6383b4a4e7b6b62ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8c347e1af7a4b6383b4a4e7b6b62ce0" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 35493 } } }
I20240502 14:13:29.424772 7017 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:29.424810 7016 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:29.426155 7029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:13:29.430488 7029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:13:29.438349 7029 catalog_manager.cc:1347] Generated new cluster ID: f0fcefa23519496495f6faae6ad75f9c
I20240502 14:13:29.438439 7029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:13:29.450420 7029 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:13:29.451665 7029 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:13:29.462323 7029 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f8c347e1af7a4b6383b4a4e7b6b62ce0: Generated new TSK 0
I20240502 14:13:29.463029 7029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:13:29.474351 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.193:0
--local_ip_for_outbound_sockets=127.6.196.193
--webserver_interface=127.6.196.193
--webserver_port=0
--tserver_master_addrs=127.6.196.254:35493
--builtin_ntp_servers=127.6.196.212:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240502 14:13:29.782596 7037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:29.782826 7037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:29.782908 7037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:29.789270 7037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:29.789429 7037 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.193
I20240502 14:13:29.797188 7037 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.196.193
--webserver_port=0
--tserver_master_addrs=127.6.196.254:35493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.193
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:29.798318 7037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:29.799758 7037 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:29.809923 7042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:29.810113 7043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:29.810360 7045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:29.810711 7037 server_base.cc:1034] running on GCE node
I20240502 14:13:29.811256 7037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:29.812077 7037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:29.813293 7037 hybrid_clock.cc:648] HybridClock initialized: now 1714659209813245 us; error 32 us; skew 500 ppm
I20240502 14:13:29.813529 7037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:29.815474 7037 webserver.cc:460] Webserver started at http://127.6.196.193:34543/ using document root <none> and password file <none>
I20240502 14:13:29.816115 7037 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:29.816231 7037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:29.816500 7037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:29.818791 7037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1aff509cf2e14b71a26a352730696ecc"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-kjf7"
I20240502 14:13:29.819442 7037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1aff509cf2e14b71a26a352730696ecc"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-kjf7"
I20240502 14:13:29.823962 7037 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 14:13:29.826709 7052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:29.827731 7037 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20240502 14:13:29.827862 7037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1aff509cf2e14b71a26a352730696ecc"
format_stamp: "Formatted at 2024-05-02 14:13:29 on dist-test-slave-kjf7"
I20240502 14:13:29.828006 7037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:29.842087 7037 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:29.842659 7037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:29.842896 7037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:29.843592 7037 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:29.844748 7037 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:29.844815 7037 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:29.844936 7037 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:29.844985 7037 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:29.872655 7037 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.193:40719
I20240502 14:13:29.872699 7170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.193:40719 every 8 connection(s)
I20240502 14:13:29.873654 7037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:13:29.881664 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 7037
I20240502 14:13:29.881891 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:13:29.885448 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.194:0
--local_ip_for_outbound_sockets=127.6.196.194
--webserver_interface=127.6.196.194
--webserver_port=0
--tserver_master_addrs=127.6.196.254:35493
--builtin_ntp_servers=127.6.196.212:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 14:13:29.886324 7171 heartbeater.cc:347] Connected to a master server at 127.6.196.254:35493
I20240502 14:13:29.886543 7171 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:29.887154 7171 heartbeater.cc:511] Master 127.6.196.254:35493 requested a full tablet report, sending...
I20240502 14:13:29.888859 6974 ts_manager.cc:194] Registered new tserver with Master: 1aff509cf2e14b71a26a352730696ecc (127.6.196.193:40719)
I20240502 14:13:29.890854 6974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.193:55425
W20240502 14:13:30.173923 7174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:30.174165 7174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:30.174254 7174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:30.181571 7174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:30.181733 7174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.194
I20240502 14:13:30.189819 7174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.196.194
--webserver_port=0
--tserver_master_addrs=127.6.196.254:35493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.194
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:30.191007 7174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:30.192690 7174 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:30.204125 7179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:30.204071 7180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:30.204372 7182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:30.204798 7174 server_base.cc:1034] running on GCE node
I20240502 14:13:30.205327 7174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:30.206112 7174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:30.207320 7174 hybrid_clock.cc:648] HybridClock initialized: now 1714659210207252 us; error 50 us; skew 500 ppm
I20240502 14:13:30.207576 7174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:30.209589 7174 webserver.cc:460] Webserver started at http://127.6.196.194:43659/ using document root <none> and password file <none>
I20240502 14:13:30.210201 7174 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:30.210322 7174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:30.210592 7174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:30.212893 7174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "76d7e0e908fe4c0987b91c3cb26afac1"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-kjf7"
I20240502 14:13:30.213603 7174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "76d7e0e908fe4c0987b91c3cb26afac1"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-kjf7"
I20240502 14:13:30.218178 7174 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:13:30.221297 7188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:30.222429 7174 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:13:30.222553 7174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "76d7e0e908fe4c0987b91c3cb26afac1"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-kjf7"
I20240502 14:13:30.222709 7174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:30.242408 7174 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:30.242993 7174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:30.243221 7174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:30.243954 7174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:30.245131 7174 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:30.245193 7174 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:30.245317 7174 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:30.245365 7174 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:30.272689 7174 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.194:37847
I20240502 14:13:30.272722 7303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.194:37847 every 8 connection(s)
I20240502 14:13:30.273691 7174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:13:30.280231 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 7174
I20240502 14:13:30.280395 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:13:30.283653 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.195:0
--local_ip_for_outbound_sockets=127.6.196.195
--webserver_interface=127.6.196.195
--webserver_port=0
--tserver_master_addrs=127.6.196.254:35493
--builtin_ntp_servers=127.6.196.212:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 14:13:30.284922 7304 heartbeater.cc:347] Connected to a master server at 127.6.196.254:35493
I20240502 14:13:30.285207 7304 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:30.286111 7304 heartbeater.cc:511] Master 127.6.196.254:35493 requested a full tablet report, sending...
I20240502 14:13:30.287344 6974 ts_manager.cc:194] Registered new tserver with Master: 76d7e0e908fe4c0987b91c3cb26afac1 (127.6.196.194:37847)
I20240502 14:13:30.287982 6974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.194:34207
W20240502 14:13:30.578202 7307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:30.578466 7307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:30.578588 7307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:30.585804 7307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:30.585976 7307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.195
I20240502 14:13:30.594112 7307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.196.195
--webserver_port=0
--tserver_master_addrs=127.6.196.254:35493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.195
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:30.595198 7307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:30.596776 7307 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:30.607514 7313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:30.607615 7312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:30.607944 7315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:30.608284 7307 server_base.cc:1034] running on GCE node
I20240502 14:13:30.608788 7307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:30.609534 7307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:30.610764 7307 hybrid_clock.cc:648] HybridClock initialized: now 1714659210610712 us; error 23 us; skew 500 ppm
I20240502 14:13:30.611060 7307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:30.613090 7307 webserver.cc:460] Webserver started at http://127.6.196.195:44223/ using document root <none> and password file <none>
I20240502 14:13:30.613711 7307 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:30.613821 7307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:30.614082 7307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:30.616314 7307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f7821e14c17b4b98be97b73a7c16d08e"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-kjf7"
I20240502 14:13:30.616961 7307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f7821e14c17b4b98be97b73a7c16d08e"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-kjf7"
I20240502 14:13:30.621613 7307 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:13:30.624375 7321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:30.625524 7307 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240502 14:13:30.625648 7307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f7821e14c17b4b98be97b73a7c16d08e"
format_stamp: "Formatted at 2024-05-02 14:13:30 on dist-test-slave-kjf7"
I20240502 14:13:30.625779 7307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:30.645714 7307 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:30.646322 7307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:30.646549 7307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:30.647292 7307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:30.648520 7307 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:30.648579 7307 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:30.648679 7307 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:30.648720 7307 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:30.676832 7307 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.195:45421
I20240502 14:13:30.676882 7434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.195:45421 every 8 connection(s)
I20240502 14:13:30.677870 7307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:13:30.678282 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 7307
I20240502 14:13:30.678467 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:13:30.682029 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.196:0
--local_ip_for_outbound_sockets=127.6.196.196
--webserver_interface=127.6.196.196
--webserver_port=0
--tserver_master_addrs=127.6.196.254:35493
--builtin_ntp_servers=127.6.196.212:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 14:13:30.691442 7435 heartbeater.cc:347] Connected to a master server at 127.6.196.254:35493
I20240502 14:13:30.691699 7435 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:30.692365 7435 heartbeater.cc:511] Master 127.6.196.254:35493 requested a full tablet report, sending...
I20240502 14:13:30.693580 6974 ts_manager.cc:194] Registered new tserver with Master: f7821e14c17b4b98be97b73a7c16d08e (127.6.196.195:45421)
I20240502 14:13:30.694734 6974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.195:51075
I20240502 14:13:30.893270 7171 heartbeater.cc:503] Master 127.6.196.254:35493 was elected leader, sending a full tablet report...
W20240502 14:13:30.976845 7438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:30.977089 7438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:30.977201 7438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:30.983901 7438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:30.984076 7438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.196
I20240502 14:13:30.991847 7438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.196.196
--webserver_port=0
--tserver_master_addrs=127.6.196.254:35493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.196
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:30.992964 7438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:30.994509 7438 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:31.004874 7443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:31.005198 7444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:31.005530 7446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:31.006481 7438 server_base.cc:1034] running on GCE node
I20240502 14:13:31.007040 7438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:31.007901 7438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:31.009156 7438 hybrid_clock.cc:648] HybridClock initialized: now 1714659211009103 us; error 31 us; skew 500 ppm
I20240502 14:13:31.009416 7438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:31.011482 7438 webserver.cc:460] Webserver started at http://127.6.196.196:39945/ using document root <none> and password file <none>
I20240502 14:13:31.012126 7438 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:31.012236 7438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:31.012466 7438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:31.014673 7438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "64abd247284c4ec88c3a83c766c99dcf"
format_stamp: "Formatted at 2024-05-02 14:13:31 on dist-test-slave-kjf7"
I20240502 14:13:31.015342 7438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "64abd247284c4ec88c3a83c766c99dcf"
format_stamp: "Formatted at 2024-05-02 14:13:31 on dist-test-slave-kjf7"
I20240502 14:13:31.019901 7438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20240502 14:13:31.023180 7452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:31.024330 7438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:13:31.024449 7438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "64abd247284c4ec88c3a83c766c99dcf"
format_stamp: "Formatted at 2024-05-02 14:13:31 on dist-test-slave-kjf7"
I20240502 14:13:31.024581 7438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:31.038313 7438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:31.038990 7438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:31.039232 7438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:31.040048 7438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:31.041456 7438 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:31.041536 7438 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:31.041635 7438 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:31.041687 7438 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:31.071664 7438 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.196:42647
I20240502 14:13:31.071725 7568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.196:42647 every 8 connection(s)
I20240502 14:13:31.072759 7438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:13:31.078099 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 7438
I20240502 14:13:31.078253 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:13:31.081073 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.197:0
--local_ip_for_outbound_sockets=127.6.196.197
--webserver_interface=127.6.196.197
--webserver_port=0
--tserver_master_addrs=127.6.196.254:35493
--builtin_ntp_servers=127.6.196.212:37407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240502 14:13:31.084996 7569 heartbeater.cc:347] Connected to a master server at 127.6.196.254:35493
I20240502 14:13:31.085250 7569 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:31.085873 7569 heartbeater.cc:511] Master 127.6.196.254:35493 requested a full tablet report, sending...
I20240502 14:13:31.087078 6974 ts_manager.cc:194] Registered new tserver with Master: 64abd247284c4ec88c3a83c766c99dcf (127.6.196.196:42647)
I20240502 14:13:31.088163 6974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.196:48709
I20240502 14:13:31.289903 7304 heartbeater.cc:503] Master 127.6.196.254:35493 was elected leader, sending a full tablet report...
W20240502 14:13:31.390322 7572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:31.390564 7572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:31.390698 7572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:31.397310 7572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:31.397502 7572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.197
I20240502 14:13:31.405402 7572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:37407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.196.197
--webserver_port=0
--tserver_master_addrs=127.6.196.254:35493
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.197
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:31.406459 7572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:31.407951 7572 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:31.418577 7578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:31.418697 7577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:31.418671 7580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:31.419236 7572 server_base.cc:1034] running on GCE node
I20240502 14:13:31.419776 7572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:31.420620 7572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:31.421818 7572 hybrid_clock.cc:648] HybridClock initialized: now 1714659211421748 us; error 39 us; skew 500 ppm
I20240502 14:13:31.422097 7572 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:31.424029 7572 webserver.cc:460] Webserver started at http://127.6.196.197:43249/ using document root <none> and password file <none>
I20240502 14:13:31.424587 7572 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:31.424695 7572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:31.424949 7572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:31.427264 7572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "7302787638ed4cc9bfdf4db2256a1e96"
format_stamp: "Formatted at 2024-05-02 14:13:31 on dist-test-slave-kjf7"
I20240502 14:13:31.427964 7572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "7302787638ed4cc9bfdf4db2256a1e96"
format_stamp: "Formatted at 2024-05-02 14:13:31 on dist-test-slave-kjf7"
I20240502 14:13:31.432461 7572 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20240502 14:13:31.435256 7586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:31.436359 7572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:13:31.436486 7572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "7302787638ed4cc9bfdf4db2256a1e96"
format_stamp: "Formatted at 2024-05-02 14:13:31 on dist-test-slave-kjf7"
I20240502 14:13:31.436620 7572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:31.451071 7572 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:31.451732 7572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:31.451999 7572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:31.452821 7572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:31.454159 7572 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:31.454231 7572 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:31.454453 7572 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:31.454519 7572 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:31.483705 7572 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.197:38909
I20240502 14:13:31.483767 7700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.197:38909 every 8 connection(s)
I20240502 14:13:31.484778 7572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:13:31.487112 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 7572
I20240502 14:13:31.487285 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:13:31.495337 7701 heartbeater.cc:347] Connected to a master server at 127.6.196.254:35493
I20240502 14:13:31.495574 7701 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:31.496205 7701 heartbeater.cc:511] Master 127.6.196.254:35493 requested a full tablet report, sending...
I20240502 14:13:31.497331 6974 ts_manager.cc:194] Registered new tserver with Master: 7302787638ed4cc9bfdf4db2256a1e96 (127.6.196.197:38909)
I20240502 14:13:31.498003 6974 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.197:50715
I20240502 14:13:31.502624 6931 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:13:31.520035 6974 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44484:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 14:13:31.549322 7102 tablet_service.cc:1467] Processing CreateTablet for tablet b1a2666e5a3b4b66ad2562923e860c55 (DEFAULT_TABLE table=TestTable [id=9dc530eb43804cf49c130f8f86a39d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:31.549315 7636 tablet_service.cc:1467] Processing CreateTablet for tablet b1a2666e5a3b4b66ad2562923e860c55 (DEFAULT_TABLE table=TestTable [id=9dc530eb43804cf49c130f8f86a39d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:31.550705 7102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a2666e5a3b4b66ad2562923e860c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:31.550715 7636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a2666e5a3b4b66ad2562923e860c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:31.553558 7502 tablet_service.cc:1467] Processing CreateTablet for tablet b1a2666e5a3b4b66ad2562923e860c55 (DEFAULT_TABLE table=TestTable [id=9dc530eb43804cf49c130f8f86a39d44]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:31.554662 7502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a2666e5a3b4b66ad2562923e860c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:31.563304 7719 tablet_bootstrap.cc:492] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc: Bootstrap starting.
I20240502 14:13:31.564833 7721 tablet_bootstrap.cc:492] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf: Bootstrap starting.
I20240502 14:13:31.565064 7720 tablet_bootstrap.cc:492] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96: Bootstrap starting.
I20240502 14:13:31.567770 7721 tablet_bootstrap.cc:654] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:31.568305 7719 tablet_bootstrap.cc:654] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:31.568576 7720 tablet_bootstrap.cc:654] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:31.568874 7721 log.cc:824] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:31.569502 7719 log.cc:824] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:31.569631 7720 log.cc:824] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:31.571588 7721 tablet_bootstrap.cc:492] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf: No bootstrap required, opened a new log
I20240502 14:13:31.571712 7719 tablet_bootstrap.cc:492] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc: No bootstrap required, opened a new log
I20240502 14:13:31.571867 7721 ts_tablet_manager.cc:1367] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240502 14:13:31.571944 7719 ts_tablet_manager.cc:1367] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:13:31.572302 7720 tablet_bootstrap.cc:492] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96: No bootstrap required, opened a new log
I20240502 14:13:31.572525 7720 ts_tablet_manager.cc:1367] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 14:13:31.577692 7719 raft_consensus.cc:348] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } }
I20240502 14:13:31.577994 7719 raft_consensus.cc:374] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:31.578114 7719 raft_consensus.cc:729] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aff509cf2e14b71a26a352730696ecc, State: Initialized, Role: FOLLOWER
I20240502 14:13:31.578660 7719 consensus_queue.cc:260] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } }
I20240502 14:13:31.580062 7719 ts_tablet_manager.cc:1398] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20240502 14:13:31.579977 7721 raft_consensus.cc:348] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } }
I20240502 14:13:31.580353 7721 raft_consensus.cc:374] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:31.580479 7721 raft_consensus.cc:729] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64abd247284c4ec88c3a83c766c99dcf, State: Initialized, Role: FOLLOWER
I20240502 14:13:31.581061 7721 consensus_queue.cc:260] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } }
I20240502 14:13:31.581084 7720 raft_consensus.cc:348] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } }
I20240502 14:13:31.581521 7720 raft_consensus.cc:374] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:31.581672 7720 raft_consensus.cc:729] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7302787638ed4cc9bfdf4db2256a1e96, State: Initialized, Role: FOLLOWER
I20240502 14:13:31.582206 7569 heartbeater.cc:503] Master 127.6.196.254:35493 was elected leader, sending a full tablet report...
I20240502 14:13:31.582701 7721 ts_tablet_manager.cc:1398] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20240502 14:13:31.583585 7720 consensus_queue.cc:260] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } }
I20240502 14:13:31.584542 7701 heartbeater.cc:503] Master 127.6.196.254:35493 was elected leader, sending a full tablet report...
I20240502 14:13:31.585168 7720 ts_tablet_manager.cc:1398] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20240502 14:13:31.625952 7172 tablet.cc:2369] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:13:31.696797 7435 heartbeater.cc:503] Master 127.6.196.254:35493 was elected leader, sending a full tablet report...
I20240502 14:13:31.701514 7726 raft_consensus.cc:482] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:31.701731 7726 raft_consensus.cc:504] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } }
I20240502 14:13:31.703101 7726 leader_election.cc:288] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1aff509cf2e14b71a26a352730696ecc (127.6.196.193:40719), 7302787638ed4cc9bfdf4db2256a1e96 (127.6.196.197:38909)
I20240502 14:13:31.709812 7123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a2666e5a3b4b66ad2562923e860c55" candidate_uuid: "64abd247284c4ec88c3a83c766c99dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aff509cf2e14b71a26a352730696ecc" is_pre_election: true
I20240502 14:13:31.710234 7123 raft_consensus.cc:2454] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64abd247284c4ec88c3a83c766c99dcf in term 0.
I20240502 14:13:31.710496 7656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a2666e5a3b4b66ad2562923e860c55" candidate_uuid: "64abd247284c4ec88c3a83c766c99dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7302787638ed4cc9bfdf4db2256a1e96" is_pre_election: true
I20240502 14:13:31.710770 7454 leader_election.cc:302] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aff509cf2e14b71a26a352730696ecc, 64abd247284c4ec88c3a83c766c99dcf; no voters:
I20240502 14:13:31.711024 7656 raft_consensus.cc:2454] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64abd247284c4ec88c3a83c766c99dcf in term 0.
I20240502 14:13:31.711180 7726 raft_consensus.cc:2781] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:31.711314 7726 raft_consensus.cc:482] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:31.711431 7726 raft_consensus.cc:3037] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:31.713340 7726 raft_consensus.cc:504] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } }
I20240502 14:13:31.713789 7726 leader_election.cc:288] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [CANDIDATE]: Term 1 election: Requested vote from peers 1aff509cf2e14b71a26a352730696ecc (127.6.196.193:40719), 7302787638ed4cc9bfdf4db2256a1e96 (127.6.196.197:38909)
I20240502 14:13:31.714130 7123 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a2666e5a3b4b66ad2562923e860c55" candidate_uuid: "64abd247284c4ec88c3a83c766c99dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1aff509cf2e14b71a26a352730696ecc"
I20240502 14:13:31.714192 7656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1a2666e5a3b4b66ad2562923e860c55" candidate_uuid: "64abd247284c4ec88c3a83c766c99dcf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7302787638ed4cc9bfdf4db2256a1e96"
I20240502 14:13:31.714267 7123 raft_consensus.cc:3037] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:31.714390 7656 raft_consensus.cc:3037] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:31.716248 7123 raft_consensus.cc:2454] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64abd247284c4ec88c3a83c766c99dcf in term 1.
I20240502 14:13:31.716370 7656 raft_consensus.cc:2454] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64abd247284c4ec88c3a83c766c99dcf in term 1.
I20240502 14:13:31.716596 7454 leader_election.cc:302] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1aff509cf2e14b71a26a352730696ecc, 64abd247284c4ec88c3a83c766c99dcf; no voters:
I20240502 14:13:31.716908 7726 raft_consensus.cc:2781] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:31.717344 7726 raft_consensus.cc:686] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 1 LEADER]: Becoming Leader. State: Replica: 64abd247284c4ec88c3a83c766c99dcf, State: Running, Role: LEADER
I20240502 14:13:31.717751 7726 consensus_queue.cc:237] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } }
I20240502 14:13:31.721828 6974 catalog_manager.cc:5484] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf reported cstate change: term changed from 0 to 1, leader changed from <none> to 64abd247284c4ec88c3a83c766c99dcf (127.6.196.196). New cstate: current_term: 1 leader_uuid: "64abd247284c4ec88c3a83c766c99dcf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } health_report { overall_health: HEALTHY } } }
W20240502 14:13:31.736168 7702 tablet.cc:2369] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:13:31.737577 6931 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:13:31.739631 6931 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 1aff509cf2e14b71a26a352730696ecc to finish bootstrapping
I20240502 14:13:31.747053 6931 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 76d7e0e908fe4c0987b91c3cb26afac1 to finish bootstrapping
I20240502 14:13:31.754770 6931 ts_itest-base.cc:233] Waiting for 0 tablets on tserver f7821e14c17b4b98be97b73a7c16d08e to finish bootstrapping
I20240502 14:13:31.762238 6931 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 64abd247284c4ec88c3a83c766c99dcf to finish bootstrapping
I20240502 14:13:31.768143 6931 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 7302787638ed4cc9bfdf4db2256a1e96 to finish bootstrapping
W20240502 14:13:31.824501 7570 tablet.cc:2369] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:13:32.177736 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:32.178306 7726 consensus_queue.cc:1035] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:32.181417 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:32.183522 7237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:32.187947 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:32.192186 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:32.236192 7730 consensus_queue.cc:1035] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:13:32.748232 7237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:32.752722 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:32.757726 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:32.758805 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:32.759055 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:33.317986 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:33.318346 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:33.327172 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:33.332309 7237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:33.355795 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:33.493463 7237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:33.506426 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:33.510942 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:33.511257 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:33.511981 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:33.697923 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:33.698760 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:33.699411 7237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:33.703567 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:33.709975 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:34.194952 6974 ts_manager.cc:292] Set tserver state for 1aff509cf2e14b71a26a352730696ecc to MAINTENANCE_MODE
I20240502 14:13:35.574651 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:35.574900 7237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:35.579980 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:35.586076 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:35.587220 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:35.724622 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:35.726811 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:35.735426 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:35.740820 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:35.747624 7237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:35.898631 7237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:35.902774 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:35.908303 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:35.909485 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:35.910305 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:35.971599 7523 consensus_queue.cc:237] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } }
I20240502 14:13:35.974128 7656 raft_consensus.cc:1261] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 1 FOLLOWER]: Refusing update from remote peer 64abd247284c4ec88c3a83c766c99dcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:13:35.974185 7123 raft_consensus.cc:1261] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 1 FOLLOWER]: Refusing update from remote peer 64abd247284c4ec88c3a83c766c99dcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:13:35.974632 8051 consensus_queue.cc:1035] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:13:35.974931 7941 consensus_queue.cc:1035] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:13:35.977632 8051 raft_consensus.cc:2932] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } }
I20240502 14:13:35.978025 7656 raft_consensus.cc:2932] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } }
I20240502 14:13:35.978323 7123 raft_consensus.cc:2932] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } }
I20240502 14:13:35.983777 6974 catalog_manager.cc:5484] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf reported cstate change: config changed from index -1 to 2. New cstate: current_term: 1 leader_uuid: "64abd247284c4ec88c3a83c766c99dcf" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:35.990190 7523 consensus_queue.cc:237] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: true } }
I20240502 14:13:35.992362 7656 raft_consensus.cc:1261] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 1 FOLLOWER]: Refusing update from remote peer 64abd247284c4ec88c3a83c766c99dcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:13:35.992987 7123 raft_consensus.cc:1261] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 1 FOLLOWER]: Refusing update from remote peer 64abd247284c4ec88c3a83c766c99dcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:13:35.993295 8051 consensus_queue.cc:1035] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:13:35.993772 8051 consensus_queue.cc:1035] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:13:35.995772 8054 raft_consensus.cc:2932] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 76d7e0e908fe4c0987b91c3cb26afac1 (127.6.196.194) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: true } } }
I20240502 14:13:35.997110 7123 raft_consensus.cc:2932] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 76d7e0e908fe4c0987b91c3cb26afac1 (127.6.196.194) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: true } } }
I20240502 14:13:35.999563 6959 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b1a2666e5a3b4b66ad2562923e860c55 with cas_config_opid_index 2: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240502 14:13:36.000662 7453 consensus_peers.cc:485] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf -> Peer 76d7e0e908fe4c0987b91c3cb26afac1 (127.6.196.194:37847): Couldn't send request to peer 76d7e0e908fe4c0987b91c3cb26afac1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1a2666e5a3b4b66ad2562923e860c55. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:36.001103 6974 catalog_manager.cc:5484] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf reported cstate change: config changed from index 2 to 3, NON_VOTER 76d7e0e908fe4c0987b91c3cb26afac1 (127.6.196.194) added. New cstate: current_term: 1 leader_uuid: "64abd247284c4ec88c3a83c766c99dcf" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:13:36.001360 7656 raft_consensus.cc:2932] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 76d7e0e908fe4c0987b91c3cb26afac1 (127.6.196.194) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: true } } }
I20240502 14:13:36.424611 8074 ts_tablet_manager.cc:902] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1: Initiating tablet copy from peer 64abd247284c4ec88c3a83c766c99dcf (127.6.196.196:42647)
I20240502 14:13:36.426230 8074 tablet_copy_client.cc:308] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:42647
I20240502 14:13:36.433699 7544 tablet_copy_service.cc:140] P 64abd247284c4ec88c3a83c766c99dcf: Received BeginTabletCopySession request for tablet b1a2666e5a3b4b66ad2562923e860c55 from peer 76d7e0e908fe4c0987b91c3cb26afac1 ({username='slave'} at 127.6.196.194:44975)
I20240502 14:13:36.433946 7544 tablet_copy_service.cc:161] P 64abd247284c4ec88c3a83c766c99dcf: Beginning new tablet copy session on tablet b1a2666e5a3b4b66ad2562923e860c55 from peer 76d7e0e908fe4c0987b91c3cb26afac1 at {username='slave'} at 127.6.196.194:44975: session id = 76d7e0e908fe4c0987b91c3cb26afac1-b1a2666e5a3b4b66ad2562923e860c55
I20240502 14:13:36.436635 7544 tablet_copy_source_session.cc:205] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:13:36.440982 8074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1a2666e5a3b4b66ad2562923e860c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:36.449374 8074 tablet_copy_client.cc:783] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1: tablet copy: Starting download of 0 data blocks...
I20240502 14:13:36.449667 8074 tablet_copy_client.cc:647] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1: tablet copy: Starting download of 1 WAL segments...
I20240502 14:13:36.451486 8074 tablet_copy_client.cc:523] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:13:36.454388 8074 tablet_bootstrap.cc:492] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1: Bootstrap starting.
I20240502 14:13:36.460181 8074 log.cc:824] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:36.463407 8074 tablet_bootstrap.cc:492] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:13:36.463752 8074 tablet_bootstrap.cc:492] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1: Bootstrap complete.
I20240502 14:13:36.464107 8074 ts_tablet_manager.cc:1367] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 14:13:36.468968 8074 raft_consensus.cc:348] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: true } }
I20240502 14:13:36.469297 8074 raft_consensus.cc:729] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 76d7e0e908fe4c0987b91c3cb26afac1, State: Initialized, Role: LEARNER
I20240502 14:13:36.469671 8074 consensus_queue.cc:260] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: true } }
I20240502 14:13:36.470852 8074 ts_tablet_manager.cc:1398] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1: Time spent starting tablet: real 0.007s user 0.006s sys 0.003s
I20240502 14:13:36.471627 7544 tablet_copy_service.cc:342] P 64abd247284c4ec88c3a83c766c99dcf: Request end of tablet copy session 76d7e0e908fe4c0987b91c3cb26afac1-b1a2666e5a3b4b66ad2562923e860c55 received from {username='slave'} at 127.6.196.194:44975
I20240502 14:13:36.471786 7544 tablet_copy_service.cc:433] P 64abd247284c4ec88c3a83c766c99dcf: ending tablet copy session 76d7e0e908fe4c0987b91c3cb26afac1-b1a2666e5a3b4b66ad2562923e860c55 on tablet b1a2666e5a3b4b66ad2562923e860c55 with peer 76d7e0e908fe4c0987b91c3cb26afac1
I20240502 14:13:37.027628 7257 raft_consensus.cc:1203] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.2->[1.3-1.3] Dedup: 1.3->[]
I20240502 14:13:37.491091 8100 raft_consensus.cc:1050] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf: attempting to promote NON_VOTER 76d7e0e908fe4c0987b91c3cb26afac1 to VOTER
I20240502 14:13:37.491724 8100 consensus_queue.cc:237] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: false } }
I20240502 14:13:37.493783 7656 raft_consensus.cc:1261] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 1 FOLLOWER]: Refusing update from remote peer 64abd247284c4ec88c3a83c766c99dcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:13:37.493896 7257 raft_consensus.cc:1261] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1 [term 1 LEARNER]: Refusing update from remote peer 64abd247284c4ec88c3a83c766c99dcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:13:37.494102 7123 raft_consensus.cc:1261] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 1 FOLLOWER]: Refusing update from remote peer 64abd247284c4ec88c3a83c766c99dcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:13:37.494343 8118 consensus_queue.cc:1035] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:13:37.494787 8099 consensus_queue.cc:1035] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:13:37.495043 8100 consensus_queue.cc:1035] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:13:37.499508 8099 raft_consensus.cc:2932] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 76d7e0e908fe4c0987b91c3cb26afac1 (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: false } } }
I20240502 14:13:37.500226 7656 raft_consensus.cc:2932] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 76d7e0e908fe4c0987b91c3cb26afac1 (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: false } } }
I20240502 14:13:37.503024 7123 raft_consensus.cc:2932] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 76d7e0e908fe4c0987b91c3cb26afac1 (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: false } } }
I20240502 14:13:37.503700 7257 raft_consensus.cc:2932] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, 76d7e0e908fe4c0987b91c3cb26afac1 (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: false } } }
I20240502 14:13:37.509620 6974 catalog_manager.cc:5484] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf reported cstate change: config changed from index 3 to 4, 76d7e0e908fe4c0987b91c3cb26afac1 (127.6.196.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "64abd247284c4ec88c3a83c766c99dcf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1aff509cf2e14b71a26a352730696ecc" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 40719 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:13:37.515854 7523 consensus_queue.cc:237] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: false } }
I20240502 14:13:37.517144 7257 raft_consensus.cc:1261] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1 [term 1 FOLLOWER]: Refusing update from remote peer 64abd247284c4ec88c3a83c766c99dcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:13:37.517612 7656 raft_consensus.cc:1261] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 1 FOLLOWER]: Refusing update from remote peer 64abd247284c4ec88c3a83c766c99dcf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:13:37.517791 8099 consensus_queue.cc:1035] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:13:37.518059 8100 consensus_queue.cc:1035] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:13:37.519810 8054 raft_consensus.cc:2932] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1aff509cf2e14b71a26a352730696ecc (127.6.196.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: false } } }
I20240502 14:13:37.520418 7656 raft_consensus.cc:2932] T b1a2666e5a3b4b66ad2562923e860c55 P 7302787638ed4cc9bfdf4db2256a1e96 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1aff509cf2e14b71a26a352730696ecc (127.6.196.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: false } } }
I20240502 14:13:37.521477 7257 raft_consensus.cc:2932] T b1a2666e5a3b4b66ad2562923e860c55 P 76d7e0e908fe4c0987b91c3cb26afac1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, VOTER 1aff509cf2e14b71a26a352730696ecc (127.6.196.193) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: false } } }
I20240502 14:13:37.522637 6959 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b1a2666e5a3b4b66ad2562923e860c55 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:13:37.523564 6971 catalog_manager.cc:5484] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf reported cstate change: config changed from index 4 to 5, VOTER 1aff509cf2e14b71a26a352730696ecc (127.6.196.193) evicted. New cstate: current_term: 1 leader_uuid: "64abd247284c4ec88c3a83c766c99dcf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7302787638ed4cc9bfdf4db2256a1e96" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 38909 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "64abd247284c4ec88c3a83c766c99dcf" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 42647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76d7e0e908fe4c0987b91c3cb26afac1" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 37847 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:13:37.527302 7102 tablet_service.cc:1514] Processing DeleteTablet for tablet b1a2666e5a3b4b66ad2562923e860c55 with delete_type TABLET_DATA_TOMBSTONED (TS 1aff509cf2e14b71a26a352730696ecc not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57716
I20240502 14:13:37.528174 8128 tablet_replica.cc:330] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc: stopping tablet replica
I20240502 14:13:37.528676 8128 raft_consensus.cc:2229] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:13:37.529127 8128 raft_consensus.cc:2258] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:13:37.530443 8128 ts_tablet_manager.cc:1875] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:13:37.534758 8128 ts_tablet_manager.cc:1888] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240502 14:13:37.534879 8128 log.cc:1196] T b1a2666e5a3b4b66ad2562923e860c55 P 1aff509cf2e14b71a26a352730696ecc: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/b1a2666e5a3b4b66ad2562923e860c55
I20240502 14:13:37.535605 6959 catalog_manager.cc:4830] TS 1aff509cf2e14b71a26a352730696ecc (127.6.196.193:40719): tablet b1a2666e5a3b4b66ad2562923e860c55 (table TestTable [id=9dc530eb43804cf49c130f8f86a39d44]) successfully deleted
I20240502 14:13:37.976253 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:37.976284 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:37.979877 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:37.981914 7237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:37.982108 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:38.119518 7237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:38.123241 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:38.124045 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:38.126053 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:38.139148 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:38.252877 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:38.257906 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:38.258692 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:38.271354 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:38.282570 7237 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:38.436810 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 7174
I20240502 14:13:38.438184 7299 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:38.541234 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 7174
W20240502 14:13:38.559260 6959 connection.cc:535] server connection from 127.6.196.194:34207 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:13:38.708195 7453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.196.194:37847: connect: Connection refused (error 111)
W20240502 14:13:38.709549 7453 consensus_peers.cc:485] T b1a2666e5a3b4b66ad2562923e860c55 P 64abd247284c4ec88c3a83c766c99dcf -> Peer 76d7e0e908fe4c0987b91c3cb26afac1 (127.6.196.194:37847): Couldn't send request to peer 76d7e0e908fe4c0987b91c3cb26afac1. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.194:37847: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:38.973697 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:38.973843 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:38.975216 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:38.976298 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:39.088500 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:39.089148 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:39.119727 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:39.125170 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:39.260993 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:39.263208 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:39.263208 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:39.264523 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:39.399570 7102 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:39.410933 7636 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:39.413664 7369 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:39.419225 7502 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:39.548907 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 7037
I20240502 14:13:39.550438 7165 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:39.656579 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 7037
I20240502 14:13:39.676057 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 7307
I20240502 14:13:39.676921 7429 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:39.776381 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 7307
W20240502 14:13:39.794306 6961 connection.cc:535] server connection from 127.6.196.195:51075 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:13:39.794569 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 7438
I20240502 14:13:39.795322 7563 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:39.901885 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 7438
I20240502 14:13:39.921962 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 7572
I20240502 14:13:39.922888 7695 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:40.024295 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 7572
I20240502 14:13:40.044485 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 6943
I20240502 14:13:40.045471 7005 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:40.169517 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 6943
2024-05-02T14:13:40Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (11237 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (11237 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240502 14:13:40.213212 6931 test_util.cc:268] Using random seed: 901467577
I20240502 14:13:40.214668 6931 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:13:40.214730 6931 ts_itest-base.cc:113] --------------
I20240502 14:13:40.214777 6931 ts_itest-base.cc:114] 5 tablet servers
I20240502 14:13:40.214840 6931 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:13:40.214903 6931 ts_itest-base.cc:116] --------------
2024-05-02T14:13:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:13:40Z Disabled control of system clock
I20240502 14:13:40.234766 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.196.254:40559
--webserver_interface=127.6.196.254
--webserver_port=0
--builtin_ntp_servers=127.6.196.212:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.196.254:40559
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 14:13:40.518939 8368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:40.519213 8368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:40.519327 8368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:40.525902 8368 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:40.526008 8368 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:13:40.526072 8368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:40.526140 8368 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:13:40.526196 8368 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:13:40.534618 8368 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.196.254:40559
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.196.254:40559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.196.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:40.535836 8368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:40.537467 8368 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:40.547713 8374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:40.548204 8373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:40.548223 8376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:40.548816 8368 server_base.cc:1034] running on GCE node
I20240502 14:13:40.549559 8368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:40.550801 8368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:40.551996 8368 hybrid_clock.cc:648] HybridClock initialized: now 1714659220551948 us; error 33 us; skew 500 ppm
I20240502 14:13:40.552311 8368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:40.554288 8368 webserver.cc:460] Webserver started at http://127.6.196.254:36507/ using document root <none> and password file <none>
I20240502 14:13:40.554884 8368 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:40.554980 8368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:40.555284 8368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:40.557586 8368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7a38adbd1091417ea32e8a962b1b99c5"
format_stamp: "Formatted at 2024-05-02 14:13:40 on dist-test-slave-kjf7"
I20240502 14:13:40.558275 8368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7a38adbd1091417ea32e8a962b1b99c5"
format_stamp: "Formatted at 2024-05-02 14:13:40 on dist-test-slave-kjf7"
I20240502 14:13:40.562773 8368 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:13:40.565511 8382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:40.567423 8368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:13:40.567571 8368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7a38adbd1091417ea32e8a962b1b99c5"
format_stamp: "Formatted at 2024-05-02 14:13:40 on dist-test-slave-kjf7"
I20240502 14:13:40.567746 8368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:40.589823 8368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:40.590598 8368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:40.590934 8368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:40.607641 8368 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.254:40559
I20240502 14:13:40.607661 8433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.254:40559 every 8 connection(s)
I20240502 14:13:40.608879 8368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:13:40.612218 8434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:40.617015 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 8368
I20240502 14:13:40.617238 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:13:40.620623 8434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5: Bootstrap starting.
I20240502 14:13:40.623661 8434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:40.624536 8434 log.cc:824] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:40.627004 8434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5: No bootstrap required, opened a new log
I20240502 14:13:40.633489 8434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a38adbd1091417ea32e8a962b1b99c5" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 40559 } }
I20240502 14:13:40.633846 8434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:40.633940 8434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a38adbd1091417ea32e8a962b1b99c5, State: Initialized, Role: FOLLOWER
I20240502 14:13:40.634543 8434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a38adbd1091417ea32e8a962b1b99c5" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 40559 } }
I20240502 14:13:40.634773 8434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:40.634888 8434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:40.635048 8434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:40.653416 8434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a38adbd1091417ea32e8a962b1b99c5" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 40559 } }
I20240502 14:13:40.653918 8434 leader_election.cc:302] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a38adbd1091417ea32e8a962b1b99c5; no voters:
I20240502 14:13:40.654455 8434 leader_election.cc:288] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:40.654562 8439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:40.655048 8439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [term 1 LEADER]: Becoming Leader. State: Replica: 7a38adbd1091417ea32e8a962b1b99c5, State: Running, Role: LEADER
I20240502 14:13:40.655618 8439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a38adbd1091417ea32e8a962b1b99c5" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 40559 } }
I20240502 14:13:40.656971 8434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:13:40.658365 8441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a38adbd1091417ea32e8a962b1b99c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a38adbd1091417ea32e8a962b1b99c5" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 40559 } } }
I20240502 14:13:40.658653 8441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:40.658385 8440 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a38adbd1091417ea32e8a962b1b99c5. Latest consensus state: current_term: 1 leader_uuid: "7a38adbd1091417ea32e8a962b1b99c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a38adbd1091417ea32e8a962b1b99c5" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 40559 } } }
I20240502 14:13:40.659118 8440 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:40.661701 8446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:13:40.665477 8446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:13:40.673362 8446 catalog_manager.cc:1347] Generated new cluster ID: ff9ae5cd1cb5484d9f2c41fc66602105
I20240502 14:13:40.673444 8446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:13:40.689811 8446 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:13:40.691195 8446 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:13:40.704941 8446 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7a38adbd1091417ea32e8a962b1b99c5: Generated new TSK 0
I20240502 14:13:40.705758 8446 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:13:40.713008 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.193:0
--local_ip_for_outbound_sockets=127.6.196.193
--webserver_interface=127.6.196.193
--webserver_port=0
--tserver_master_addrs=127.6.196.254:40559
--builtin_ntp_servers=127.6.196.212:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:13:41.021978 8458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:41.022243 8458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:41.022375 8458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:41.028999 8458 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:41.029109 8458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:41.029265 8458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.193
I20240502 14:13:41.037174 8458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.196.193
--webserver_port=0
--tserver_master_addrs=127.6.196.254:40559
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.193
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:41.038331 8458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:41.039968 8458 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:41.050228 8463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:41.050804 8464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:41.051414 8466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:41.051616 8458 server_base.cc:1034] running on GCE node
I20240502 14:13:41.052111 8458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:41.052927 8458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:41.054102 8458 hybrid_clock.cc:648] HybridClock initialized: now 1714659221054062 us; error 37 us; skew 500 ppm
I20240502 14:13:41.054354 8458 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:41.056195 8458 webserver.cc:460] Webserver started at http://127.6.196.193:38433/ using document root <none> and password file <none>
I20240502 14:13:41.056793 8458 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:41.056898 8458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:41.057207 8458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:41.059373 8458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b34323ca290846918a00f972d8adf441"
format_stamp: "Formatted at 2024-05-02 14:13:41 on dist-test-slave-kjf7"
I20240502 14:13:41.060024 8458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b34323ca290846918a00f972d8adf441"
format_stamp: "Formatted at 2024-05-02 14:13:41 on dist-test-slave-kjf7"
I20240502 14:13:41.064838 8458 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20240502 14:13:41.067740 8472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:41.068796 8458 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:13:41.068902 8458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b34323ca290846918a00f972d8adf441"
format_stamp: "Formatted at 2024-05-02 14:13:41 on dist-test-slave-kjf7"
I20240502 14:13:41.069061 8458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:41.108947 8458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:41.109706 8458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:41.109934 8458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:41.110729 8458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:41.112049 8458 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:41.112133 8458 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:41.112210 8458 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:41.112262 8458 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:41.141010 8458 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.193:33677
I20240502 14:13:41.141086 8585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.193:33677 every 8 connection(s)
I20240502 14:13:41.142192 8458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:13:41.149413 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 8458
I20240502 14:13:41.149566 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:13:41.152730 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.194:0
--local_ip_for_outbound_sockets=127.6.196.194
--webserver_interface=127.6.196.194
--webserver_port=0
--tserver_master_addrs=127.6.196.254:40559
--builtin_ntp_servers=127.6.196.212:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:41.153595 8586 heartbeater.cc:347] Connected to a master server at 127.6.196.254:40559
I20240502 14:13:41.153831 8586 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:41.154536 8586 heartbeater.cc:511] Master 127.6.196.254:40559 requested a full tablet report, sending...
I20240502 14:13:41.156370 8399 ts_manager.cc:194] Registered new tserver with Master: b34323ca290846918a00f972d8adf441 (127.6.196.193:33677)
I20240502 14:13:41.157997 8399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.193:51081
W20240502 14:13:41.459686 8590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:41.459955 8590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:41.460085 8590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:41.467674 8590 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:41.467764 8590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:41.467912 8590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.194
I20240502 14:13:41.477913 8590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.196.194
--webserver_port=0
--tserver_master_addrs=127.6.196.254:40559
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.194
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:41.479125 8590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:41.480697 8590 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:41.491389 8596 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:41.492039 8595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:41.492604 8598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:41.492937 8590 server_base.cc:1034] running on GCE node
I20240502 14:13:41.493476 8590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:41.494235 8590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:41.495429 8590 hybrid_clock.cc:648] HybridClock initialized: now 1714659221495390 us; error 33 us; skew 500 ppm
I20240502 14:13:41.495711 8590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:41.497752 8590 webserver.cc:460] Webserver started at http://127.6.196.194:38305/ using document root <none> and password file <none>
I20240502 14:13:41.498440 8590 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:41.498543 8590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:41.498862 8590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:41.501150 8590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "499a45fe4b0149a1b597b60d97e910e8"
format_stamp: "Formatted at 2024-05-02 14:13:41 on dist-test-slave-kjf7"
I20240502 14:13:41.501857 8590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "499a45fe4b0149a1b597b60d97e910e8"
format_stamp: "Formatted at 2024-05-02 14:13:41 on dist-test-slave-kjf7"
I20240502 14:13:41.506325 8590 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:13:41.509358 8604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:41.510398 8590 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240502 14:13:41.510519 8590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "499a45fe4b0149a1b597b60d97e910e8"
format_stamp: "Formatted at 2024-05-02 14:13:41 on dist-test-slave-kjf7"
I20240502 14:13:41.510684 8590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:41.542891 8590 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:41.543578 8590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:41.543819 8590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:41.544605 8590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:41.545861 8590 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:41.545948 8590 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:41.546064 8590 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:41.546130 8590 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:41.577086 8590 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.194:45041
I20240502 14:13:41.577145 8716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.194:45041 every 8 connection(s)
I20240502 14:13:41.578264 8590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:13:41.580076 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 8590
I20240502 14:13:41.580277 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:13:41.583731 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.195:0
--local_ip_for_outbound_sockets=127.6.196.195
--webserver_interface=127.6.196.195
--webserver_port=0
--tserver_master_addrs=127.6.196.254:40559
--builtin_ntp_servers=127.6.196.212:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:41.591131 8717 heartbeater.cc:347] Connected to a master server at 127.6.196.254:40559
I20240502 14:13:41.591430 8717 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:41.592376 8717 heartbeater.cc:511] Master 127.6.196.254:40559 requested a full tablet report, sending...
I20240502 14:13:41.593767 8399 ts_manager.cc:194] Registered new tserver with Master: 499a45fe4b0149a1b597b60d97e910e8 (127.6.196.194:45041)
I20240502 14:13:41.594482 8399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.194:50491
W20240502 14:13:41.886632 8720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:41.886925 8720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:41.887005 8720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:41.893931 8720 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:41.894027 8720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:41.894135 8720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.195
I20240502 14:13:41.902402 8720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.196.195
--webserver_port=0
--tserver_master_addrs=127.6.196.254:40559
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.195
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:41.903486 8720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:41.904946 8720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:41.916558 8725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:41.916608 8726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:41.917266 8729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:41.917873 8720 server_base.cc:1034] running on GCE node
I20240502 14:13:41.918404 8720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:41.919327 8720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:41.920543 8720 hybrid_clock.cc:648] HybridClock initialized: now 1714659221920491 us; error 41 us; skew 500 ppm
I20240502 14:13:41.920884 8720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:41.923631 8720 webserver.cc:460] Webserver started at http://127.6.196.195:36339/ using document root <none> and password file <none>
I20240502 14:13:41.924319 8720 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:41.924454 8720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:41.924795 8720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:41.927206 8720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8f5bc88bbf39491991d6d2b06227cc9d"
format_stamp: "Formatted at 2024-05-02 14:13:41 on dist-test-slave-kjf7"
I20240502 14:13:41.927953 8720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8f5bc88bbf39491991d6d2b06227cc9d"
format_stamp: "Formatted at 2024-05-02 14:13:41 on dist-test-slave-kjf7"
I20240502 14:13:41.932696 8720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:13:41.935765 8735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:41.936882 8720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:13:41.937023 8720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8f5bc88bbf39491991d6d2b06227cc9d"
format_stamp: "Formatted at 2024-05-02 14:13:41 on dist-test-slave-kjf7"
I20240502 14:13:41.937199 8720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:41.950321 8720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:41.951006 8720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:41.951247 8720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:41.952088 8720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:41.953552 8720 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:41.953646 8720 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:41.953758 8720 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:41.953827 8720 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:41.987427 8720 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.195:38923
I20240502 14:13:41.987493 8849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.195:38923 every 8 connection(s)
I20240502 14:13:41.988760 8720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:13:41.990144 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 8720
I20240502 14:13:41.990273 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:13:41.993651 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.196:0
--local_ip_for_outbound_sockets=127.6.196.196
--webserver_interface=127.6.196.196
--webserver_port=0
--tserver_master_addrs=127.6.196.254:40559
--builtin_ntp_servers=127.6.196.212:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:42.002314 8850 heartbeater.cc:347] Connected to a master server at 127.6.196.254:40559
I20240502 14:13:42.002596 8850 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:42.003381 8850 heartbeater.cc:511] Master 127.6.196.254:40559 requested a full tablet report, sending...
I20240502 14:13:42.004653 8399 ts_manager.cc:194] Registered new tserver with Master: 8f5bc88bbf39491991d6d2b06227cc9d (127.6.196.195:38923)
I20240502 14:13:42.005448 8399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.195:49387
I20240502 14:13:42.161442 8586 heartbeater.cc:503] Master 127.6.196.254:40559 was elected leader, sending a full tablet report...
W20240502 14:13:42.295070 8853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:42.295328 8853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:42.295403 8853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:42.301793 8853 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:42.301883 8853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:42.302003 8853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.196
I20240502 14:13:42.309372 8853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.196.196
--webserver_port=0
--tserver_master_addrs=127.6.196.254:40559
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.196
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:42.310473 8853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:42.312091 8853 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:42.324175 8859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:42.324159 8858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:42.324975 8861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:42.325693 8853 server_base.cc:1034] running on GCE node
I20240502 14:13:42.326241 8853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:42.327039 8853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:42.328248 8853 hybrid_clock.cc:648] HybridClock initialized: now 1714659222328202 us; error 33 us; skew 500 ppm
I20240502 14:13:42.328542 8853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:42.330559 8853 webserver.cc:460] Webserver started at http://127.6.196.196:46371/ using document root <none> and password file <none>
I20240502 14:13:42.331218 8853 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:42.331318 8853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:42.331602 8853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:42.334054 8853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "581e9b34c40c477e8231720a10773788"
format_stamp: "Formatted at 2024-05-02 14:13:42 on dist-test-slave-kjf7"
I20240502 14:13:42.334689 8853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "581e9b34c40c477e8231720a10773788"
format_stamp: "Formatted at 2024-05-02 14:13:42 on dist-test-slave-kjf7"
I20240502 14:13:42.339116 8853 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:13:42.342195 8867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:42.343410 8853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:13:42.343544 8853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "581e9b34c40c477e8231720a10773788"
format_stamp: "Formatted at 2024-05-02 14:13:42 on dist-test-slave-kjf7"
I20240502 14:13:42.343705 8853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:42.363085 8853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:42.363901 8853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:42.364140 8853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:42.364965 8853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:42.366395 8853 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:42.366487 8853 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:42.366597 8853 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:42.366667 8853 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:42.398224 8853 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.196:41541
I20240502 14:13:42.398263 8980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.196:41541 every 8 connection(s)
I20240502 14:13:42.399463 8853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:13:42.401512 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 8853
I20240502 14:13:42.401710 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:13:42.404708 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.197:0
--local_ip_for_outbound_sockets=127.6.196.197
--webserver_interface=127.6.196.197
--webserver_port=0
--tserver_master_addrs=127.6.196.254:40559
--builtin_ntp_servers=127.6.196.212:34007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:42.412258 8981 heartbeater.cc:347] Connected to a master server at 127.6.196.254:40559
I20240502 14:13:42.412518 8981 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:42.413332 8981 heartbeater.cc:511] Master 127.6.196.254:40559 requested a full tablet report, sending...
I20240502 14:13:42.414739 8399 ts_manager.cc:194] Registered new tserver with Master: 581e9b34c40c477e8231720a10773788 (127.6.196.196:41541)
I20240502 14:13:42.416013 8399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.196:53973
I20240502 14:13:42.597189 8717 heartbeater.cc:503] Master 127.6.196.254:40559 was elected leader, sending a full tablet report...
W20240502 14:13:42.723220 8984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:42.723464 8984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:42.723541 8984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:42.729974 8984 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:42.730083 8984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:42.730242 8984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.197
I20240502 14:13:42.737987 8984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:34007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.196.197
--webserver_port=0
--tserver_master_addrs=127.6.196.254:40559
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.197
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:42.739035 8984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:42.740646 8984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:42.751283 8990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:42.751873 8989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:42.752573 8992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:42.753055 8984 server_base.cc:1034] running on GCE node
I20240502 14:13:42.753603 8984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:42.754444 8984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:42.755648 8984 hybrid_clock.cc:648] HybridClock initialized: now 1714659222755601 us; error 30 us; skew 500 ppm
I20240502 14:13:42.755923 8984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:42.757865 8984 webserver.cc:460] Webserver started at http://127.6.196.197:33109/ using document root <none> and password file <none>
I20240502 14:13:42.758491 8984 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:42.758581 8984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:42.758867 8984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:42.761126 8984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "f3f7fa93b0d74cb0ab197d5788e116dd"
format_stamp: "Formatted at 2024-05-02 14:13:42 on dist-test-slave-kjf7"
I20240502 14:13:42.761817 8984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "f3f7fa93b0d74cb0ab197d5788e116dd"
format_stamp: "Formatted at 2024-05-02 14:13:42 on dist-test-slave-kjf7"
I20240502 14:13:42.766328 8984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:13:42.769405 8998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:42.770490 8984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:13:42.770634 8984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "f3f7fa93b0d74cb0ab197d5788e116dd"
format_stamp: "Formatted at 2024-05-02 14:13:42 on dist-test-slave-kjf7"
I20240502 14:13:42.770798 8984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:42.802417 8984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:42.803308 8984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:42.803555 8984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:42.804514 8984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:42.805811 8984 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:42.805886 8984 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:42.805959 8984 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:42.806005 8984 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:42.835100 8984 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.197:43719
I20240502 14:13:42.835158 9111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.197:43719 every 8 connection(s)
I20240502 14:13:42.836189 8984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:13:42.842550 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 8984
I20240502 14:13:42.842702 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:13:42.847502 9112 heartbeater.cc:347] Connected to a master server at 127.6.196.254:40559
I20240502 14:13:42.847776 9112 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:42.848424 9112 heartbeater.cc:511] Master 127.6.196.254:40559 requested a full tablet report, sending...
I20240502 14:13:42.849643 8399 ts_manager.cc:194] Registered new tserver with Master: f3f7fa93b0d74cb0ab197d5788e116dd (127.6.196.197:43719)
I20240502 14:13:42.850387 8399 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.197:59671
I20240502 14:13:42.857704 6931 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240502 14:13:42.871521 8399 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45564:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 14:13:42.913018 8652 tablet_service.cc:1467] Processing CreateTablet for tablet 9666aacdfb994ec59309f06ca7cc2bb5 (DEFAULT_TABLE table=TestTable [id=44ae70a88c5a42a2804f1e06de853065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:42.914623 8652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9666aacdfb994ec59309f06ca7cc2bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:42.922797 9047 tablet_service.cc:1467] Processing CreateTablet for tablet 9666aacdfb994ec59309f06ca7cc2bb5 (DEFAULT_TABLE table=TestTable [id=44ae70a88c5a42a2804f1e06de853065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:42.924033 9047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9666aacdfb994ec59309f06ca7cc2bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:42.924736 8521 tablet_service.cc:1467] Processing CreateTablet for tablet 9666aacdfb994ec59309f06ca7cc2bb5 (DEFAULT_TABLE table=TestTable [id=44ae70a88c5a42a2804f1e06de853065]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:42.925971 8521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9666aacdfb994ec59309f06ca7cc2bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:42.929610 9131 tablet_bootstrap.cc:492] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8: Bootstrap starting.
I20240502 14:13:42.933028 9131 tablet_bootstrap.cc:654] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:42.934223 9131 log.cc:824] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:42.934387 9132 tablet_bootstrap.cc:492] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd: Bootstrap starting.
I20240502 14:13:42.937964 9132 tablet_bootstrap.cc:654] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:42.938201 9134 tablet_bootstrap.cc:492] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441: Bootstrap starting.
I20240502 14:13:42.939064 9132 log.cc:824] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:42.939469 9131 tablet_bootstrap.cc:492] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8: No bootstrap required, opened a new log
I20240502 14:13:42.939819 9131 ts_tablet_manager.cc:1367] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8: Time spent bootstrapping tablet: real 0.010s user 0.010s sys 0.000s
I20240502 14:13:42.941202 9132 tablet_bootstrap.cc:492] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd: No bootstrap required, opened a new log
I20240502 14:13:42.941442 9132 ts_tablet_manager.cc:1367] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240502 14:13:42.941709 9134 tablet_bootstrap.cc:654] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:42.942828 9134 log.cc:824] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:42.945159 9134 tablet_bootstrap.cc:492] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441: No bootstrap required, opened a new log
I20240502 14:13:42.945402 9134 ts_tablet_manager.cc:1367] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20240502 14:13:42.947036 9131 raft_consensus.cc:348] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } }
I20240502 14:13:42.947525 9131 raft_consensus.cc:374] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:42.947633 9131 raft_consensus.cc:729] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 499a45fe4b0149a1b597b60d97e910e8, State: Initialized, Role: FOLLOWER
I20240502 14:13:42.948261 9131 consensus_queue.cc:260] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } }
I20240502 14:13:42.948648 9132 raft_consensus.cc:348] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } }
I20240502 14:13:42.949036 9132 raft_consensus.cc:374] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:42.949138 9132 raft_consensus.cc:729] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3f7fa93b0d74cb0ab197d5788e116dd, State: Initialized, Role: FOLLOWER
I20240502 14:13:42.949728 9132 consensus_queue.cc:260] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } }
I20240502 14:13:42.950898 9112 heartbeater.cc:503] Master 127.6.196.254:40559 was elected leader, sending a full tablet report...
I20240502 14:13:42.951246 9132 ts_tablet_manager.cc:1398] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20240502 14:13:42.952647 9131 ts_tablet_manager.cc:1398] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20240502 14:13:42.953537 9134 raft_consensus.cc:348] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } }
I20240502 14:13:42.953899 9134 raft_consensus.cc:374] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:42.954005 9134 raft_consensus.cc:729] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b34323ca290846918a00f972d8adf441, State: Initialized, Role: FOLLOWER
I20240502 14:13:42.954560 9134 consensus_queue.cc:260] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } }
I20240502 14:13:42.957695 9134 ts_tablet_manager.cc:1398] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20240502 14:13:43.007577 8850 heartbeater.cc:503] Master 127.6.196.254:40559 was elected leader, sending a full tablet report...
I20240502 14:13:43.009104 9139 raft_consensus.cc:482] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:43.009337 9139 raft_consensus.cc:504] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } }
I20240502 14:13:43.010910 9139 leader_election.cc:288] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 499a45fe4b0149a1b597b60d97e910e8 (127.6.196.194:45041), f3f7fa93b0d74cb0ab197d5788e116dd (127.6.196.197:43719)
I20240502 14:13:43.019645 8672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9666aacdfb994ec59309f06ca7cc2bb5" candidate_uuid: "b34323ca290846918a00f972d8adf441" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499a45fe4b0149a1b597b60d97e910e8" is_pre_election: true
I20240502 14:13:43.019645 9067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9666aacdfb994ec59309f06ca7cc2bb5" candidate_uuid: "b34323ca290846918a00f972d8adf441" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" is_pre_election: true
I20240502 14:13:43.020175 8672 raft_consensus.cc:2454] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b34323ca290846918a00f972d8adf441 in term 0.
I20240502 14:13:43.020174 9067 raft_consensus.cc:2454] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b34323ca290846918a00f972d8adf441 in term 0.
I20240502 14:13:43.020805 8476 leader_election.cc:302] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499a45fe4b0149a1b597b60d97e910e8, b34323ca290846918a00f972d8adf441; no voters:
I20240502 14:13:43.021198 9139 raft_consensus.cc:2781] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:13:43.021373 9139 raft_consensus.cc:482] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:43.021482 9139 raft_consensus.cc:3037] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:13:43.080636 8718 tablet.cc:2369] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:13:43.087543 9113 tablet.cc:2369] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:13:43.108121 9139 raft_consensus.cc:504] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } }
I20240502 14:13:43.108774 9139 leader_election.cc:288] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [CANDIDATE]: Term 1 election: Requested vote from peers 499a45fe4b0149a1b597b60d97e910e8 (127.6.196.194:45041), f3f7fa93b0d74cb0ab197d5788e116dd (127.6.196.197:43719)
I20240502 14:13:43.109330 8672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9666aacdfb994ec59309f06ca7cc2bb5" candidate_uuid: "b34323ca290846918a00f972d8adf441" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "499a45fe4b0149a1b597b60d97e910e8"
I20240502 14:13:43.109390 9067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9666aacdfb994ec59309f06ca7cc2bb5" candidate_uuid: "b34323ca290846918a00f972d8adf441" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd"
I20240502 14:13:43.109512 8672 raft_consensus.cc:3037] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:43.109609 9067 raft_consensus.cc:3037] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:43.111979 8672 raft_consensus.cc:2454] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b34323ca290846918a00f972d8adf441 in term 1.
I20240502 14:13:43.112385 8476 leader_election.cc:302] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 499a45fe4b0149a1b597b60d97e910e8, b34323ca290846918a00f972d8adf441; no voters:
I20240502 14:13:43.112660 9139 raft_consensus.cc:2781] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:43.113821 9139 raft_consensus.cc:686] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [term 1 LEADER]: Becoming Leader. State: Replica: b34323ca290846918a00f972d8adf441, State: Running, Role: LEADER
I20240502 14:13:43.114352 9139 consensus_queue.cc:237] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } }
I20240502 14:13:43.118476 8399 catalog_manager.cc:5484] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441 reported cstate change: term changed from 0 to 1, leader changed from <none> to b34323ca290846918a00f972d8adf441 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "b34323ca290846918a00f972d8adf441" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:13:43.144507 6931 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
W20240502 14:13:43.144842 8587 tablet.cc:2369] T 9666aacdfb994ec59309f06ca7cc2bb5 P b34323ca290846918a00f972d8adf441: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:13:43.146544 6931 ts_itest-base.cc:233] Waiting for 1 tablets on tserver b34323ca290846918a00f972d8adf441 to finish bootstrapping
I20240502 14:13:43.154516 6931 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 499a45fe4b0149a1b597b60d97e910e8 to finish bootstrapping
I20240502 14:13:43.160838 6931 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 8f5bc88bbf39491991d6d2b06227cc9d to finish bootstrapping
I20240502 14:13:43.168053 6931 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 581e9b34c40c477e8231720a10773788 to finish bootstrapping
I20240502 14:13:43.175485 6931 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f3f7fa93b0d74cb0ab197d5788e116dd to finish bootstrapping
I20240502 14:13:43.209657 9067 raft_consensus.cc:2454] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b34323ca290846918a00f972d8adf441 in term 1.
I20240502 14:13:43.211562 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 8458
I20240502 14:13:43.212631 8581 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:43.294152 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 8458
I20240502 14:13:43.419035 8981 heartbeater.cc:503] Master 127.6.196.254:40559 was elected leader, sending a full tablet report...
I20240502 14:13:43.757922 8652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.769099 8916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.770831 8784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.775740 9047 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.901000 8916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.901576 8652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.906275 8784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:43.907496 9047 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.061408 8784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.061520 8652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.062379 8916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.069974 9047 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.642935 8916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.654857 8652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.655438 8784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.661948 9047 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.781528 8652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.789337 8784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.795341 8916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.796386 9047 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.900141 9321 raft_consensus.cc:482] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:13:44.900319 9321 raft_consensus.cc:504] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } }
I20240502 14:13:44.902209 9321 leader_election.cc:288] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b34323ca290846918a00f972d8adf441 (127.6.196.193:33677), f3f7fa93b0d74cb0ab197d5788e116dd (127.6.196.197:43719)
W20240502 14:13:44.903782 8606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.196.193:33677: connect: Connection refused (error 111)
W20240502 14:13:44.905385 8606 leader_election.cc:334] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b34323ca290846918a00f972d8adf441 (127.6.196.193:33677): Network error: Client connection negotiation failed: client connection to 127.6.196.193:33677: connect: Connection refused (error 111)
I20240502 14:13:44.908363 9067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9666aacdfb994ec59309f06ca7cc2bb5" candidate_uuid: "499a45fe4b0149a1b597b60d97e910e8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" is_pre_election: true
I20240502 14:13:44.908571 9067 raft_consensus.cc:2454] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 499a45fe4b0149a1b597b60d97e910e8 in term 1.
I20240502 14:13:44.908968 8605 leader_election.cc:302] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 499a45fe4b0149a1b597b60d97e910e8, f3f7fa93b0d74cb0ab197d5788e116dd; no voters: b34323ca290846918a00f972d8adf441
I20240502 14:13:44.909346 9321 raft_consensus.cc:2781] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:13:44.909509 9321 raft_consensus.cc:482] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:13:44.909626 9321 raft_consensus.cc:3037] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:44.916342 9321 raft_consensus.cc:504] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } }
I20240502 14:13:44.917063 9321 leader_election.cc:288] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [CANDIDATE]: Term 2 election: Requested vote from peers b34323ca290846918a00f972d8adf441 (127.6.196.193:33677), f3f7fa93b0d74cb0ab197d5788e116dd (127.6.196.197:43719)
I20240502 14:13:44.917537 9067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9666aacdfb994ec59309f06ca7cc2bb5" candidate_uuid: "499a45fe4b0149a1b597b60d97e910e8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd"
I20240502 14:13:44.917757 9067 raft_consensus.cc:3037] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:13:44.918182 8606 leader_election.cc:334] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b34323ca290846918a00f972d8adf441 (127.6.196.193:33677): Network error: Client connection negotiation failed: client connection to 127.6.196.193:33677: connect: Connection refused (error 111)
I20240502 14:13:44.953392 8652 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.953758 8916 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:44.963263 8784 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:45.017081 9067 raft_consensus.cc:2454] T 9666aacdfb994ec59309f06ca7cc2bb5 P f3f7fa93b0d74cb0ab197d5788e116dd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 499a45fe4b0149a1b597b60d97e910e8 in term 2.
I20240502 14:13:45.017783 8605 leader_election.cc:302] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 499a45fe4b0149a1b597b60d97e910e8, f3f7fa93b0d74cb0ab197d5788e116dd; no voters: b34323ca290846918a00f972d8adf441
I20240502 14:13:45.018222 9321 raft_consensus.cc:2781] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:45.018735 9321 raft_consensus.cc:686] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [term 2 LEADER]: Becoming Leader. State: Replica: 499a45fe4b0149a1b597b60d97e910e8, State: Running, Role: LEADER
I20240502 14:13:45.019326 9321 consensus_queue.cc:237] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } }
I20240502 14:13:45.019829 9047 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:45.024363 8399 catalog_manager.cc:5484] T 9666aacdfb994ec59309f06ca7cc2bb5 P 499a45fe4b0149a1b597b60d97e910e8 reported cstate change: term changed from 1 to 2, leader changed from b34323ca290846918a00f972d8adf441 (127.6.196.193) to 499a45fe4b0149a1b597b60d97e910e8 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "499a45fe4b0149a1b597b60d97e910e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "499a45fe4b0149a1b597b60d97e910e8" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 45041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b34323ca290846918a00f972d8adf441" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3f7fa93b0d74cb0ab197d5788e116dd" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43719 } health_report { overall_health: UNKNOWN } } }
I20240502 14:13:45.149155 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 8590
I20240502 14:13:45.151208 8711 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:45.247596 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 8590
I20240502 14:13:45.266522 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 8720
I20240502 14:13:45.267441 8845 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:45.352542 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 8720
I20240502 14:13:45.368793 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 8853
I20240502 14:13:45.369705 8975 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:45.451418 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 8853
W20240502 14:13:45.467821 8386 connection.cc:535] server connection from 127.6.196.196:53973 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:13:45.468099 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 8984
I20240502 14:13:45.468921 9106 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:45.556126 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 8984
I20240502 14:13:45.573148 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 8368
I20240502 14:13:45.574163 8429 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:45.681432 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 8368
2024-05-02T14:13:45Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (5509 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5509 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240502 14:13:45.722618 6931 test_util.cc:268] Using random seed: 906976993
I20240502 14:13:45.724334 6931 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:13:45.724395 6931 ts_itest-base.cc:113] --------------
I20240502 14:13:45.724443 6931 ts_itest-base.cc:114] 7 tablet servers
I20240502 14:13:45.724485 6931 ts_itest-base.cc:115] 1 replicas per TS
I20240502 14:13:45.724545 6931 ts_itest-base.cc:116] --------------
2024-05-02T14:13:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:13:45Z Disabled control of system clock
I20240502 14:13:45.745781 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.196.254:43585
--webserver_interface=127.6.196.254
--webserver_port=0
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.196.254:43585
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240502 14:13:46.029557 9358 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240502 14:13:46.029878 9358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:46.029980 9358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:46.030081 9358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:46.036936 9358 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:46.037055 9358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:13:46.037139 9358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:46.037199 9358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:13:46.037292 9358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:13:46.045120 9358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.196.254:43585
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.196.254:43585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.196.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:46.046160 9358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:46.047618 9358 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:46.057063 9363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.057732 9366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.058319 9364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:46.058933 9358 server_base.cc:1034] running on GCE node
I20240502 14:13:46.059515 9358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:46.060603 9358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:46.061786 9358 hybrid_clock.cc:648] HybridClock initialized: now 1714659226061720 us; error 47 us; skew 500 ppm
I20240502 14:13:46.062055 9358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:46.064002 9358 webserver.cc:460] Webserver started at http://127.6.196.254:44053/ using document root <none> and password file <none>
I20240502 14:13:46.064575 9358 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:46.064698 9358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:46.064983 9358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:46.067291 9358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "247adb9a8de14791b6b2872e02aeff89"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-kjf7"
I20240502 14:13:46.067965 9358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "247adb9a8de14791b6b2872e02aeff89"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-kjf7"
I20240502 14:13:46.072168 9358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.002s
I20240502 14:13:46.075006 9373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.076051 9358 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:13:46.076184 9358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
uuid: "247adb9a8de14791b6b2872e02aeff89"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-kjf7"
I20240502 14:13:46.076325 9358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:46.094753 9358 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:46.095350 9358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:46.095573 9358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:46.111433 9358 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.254:43585
I20240502 14:13:46.111457 9424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.254:43585 every 8 connection(s)
I20240502 14:13:46.112541 9358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:13:46.115752 9425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:46.117857 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 9358
I20240502 14:13:46.118037 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:13:46.125507 9425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89: Bootstrap starting.
I20240502 14:13:46.128281 9425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:46.129509 9425 log.cc:824] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:46.131848 9425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89: No bootstrap required, opened a new log
I20240502 14:13:46.137581 9425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247adb9a8de14791b6b2872e02aeff89" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 43585 } }
I20240502 14:13:46.137836 9425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:46.137902 9425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 247adb9a8de14791b6b2872e02aeff89, State: Initialized, Role: FOLLOWER
I20240502 14:13:46.138357 9425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247adb9a8de14791b6b2872e02aeff89" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 43585 } }
I20240502 14:13:46.138506 9425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:46.138605 9425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:46.138707 9425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:46.140550 9425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247adb9a8de14791b6b2872e02aeff89" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 43585 } }
I20240502 14:13:46.140933 9425 leader_election.cc:302] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 247adb9a8de14791b6b2872e02aeff89; no voters:
I20240502 14:13:46.141413 9425 leader_election.cc:288] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:46.141518 9430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:46.142050 9430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [term 1 LEADER]: Becoming Leader. State: Replica: 247adb9a8de14791b6b2872e02aeff89, State: Running, Role: LEADER
I20240502 14:13:46.142544 9430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247adb9a8de14791b6b2872e02aeff89" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 43585 } }
I20240502 14:13:46.143074 9425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:13:46.145012 9431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "247adb9a8de14791b6b2872e02aeff89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247adb9a8de14791b6b2872e02aeff89" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 43585 } } }
I20240502 14:13:46.145360 9431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:46.145763 9432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 247adb9a8de14791b6b2872e02aeff89. Latest consensus state: current_term: 1 leader_uuid: "247adb9a8de14791b6b2872e02aeff89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "247adb9a8de14791b6b2872e02aeff89" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 43585 } } }
I20240502 14:13:46.146062 9432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89 [sys.catalog]: This master's current role is: LEADER
I20240502 14:13:46.150132 9438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:13:46.154737 9438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:13:46.161957 9438 catalog_manager.cc:1347] Generated new cluster ID: 52a233d8d37a481bb3fa3be7f691041c
I20240502 14:13:46.162060 9438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:13:46.181299 9438 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:13:46.182698 9438 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:13:46.193503 9438 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 247adb9a8de14791b6b2872e02aeff89: Generated new TSK 0
I20240502 14:13:46.194128 9438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:13:46.206072 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.193:0
--local_ip_for_outbound_sockets=127.6.196.193
--webserver_interface=127.6.196.193
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:13:46.528875 9449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:46.529186 9449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:46.529321 9449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:46.536036 9449 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:46.536128 9449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:46.536229 9449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.193
I20240502 14:13:46.544003 9449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.196.193
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.193
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:46.545125 9449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:46.546700 9449 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:46.557567 9454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.557981 9457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.558792 9455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:46.559478 9449 server_base.cc:1034] running on GCE node
I20240502 14:13:46.560019 9449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:46.560801 9449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:46.562000 9449 hybrid_clock.cc:648] HybridClock initialized: now 1714659226561940 us; error 42 us; skew 500 ppm
I20240502 14:13:46.562297 9449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:46.564309 9449 webserver.cc:460] Webserver started at http://127.6.196.193:34337/ using document root <none> and password file <none>
I20240502 14:13:46.564918 9449 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:46.565027 9449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:46.565352 9449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:46.567481 9449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5968a421103245be945b2453edc11c01"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-kjf7"
I20240502 14:13:46.568143 9449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5968a421103245be945b2453edc11c01"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-kjf7"
I20240502 14:13:46.572495 9449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:13:46.575440 9464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.576526 9449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:13:46.576654 9449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5968a421103245be945b2453edc11c01"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-kjf7"
I20240502 14:13:46.576812 9449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:46.606189 9449 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:46.606923 9449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:46.607167 9449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:46.607957 9449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:46.609284 9449 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:46.609372 9449 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.609474 9449 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:46.609539 9449 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.638346 9449 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.193:33389
I20240502 14:13:46.638410 9577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.193:33389 every 8 connection(s)
I20240502 14:13:46.639451 9449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:13:46.644187 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 9449
I20240502 14:13:46.644416 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:13:46.647490 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.194:0
--local_ip_for_outbound_sockets=127.6.196.194
--webserver_interface=127.6.196.194
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:46.651608 9578 heartbeater.cc:347] Connected to a master server at 127.6.196.254:43585
I20240502 14:13:46.651852 9578 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:46.652562 9578 heartbeater.cc:511] Master 127.6.196.254:43585 requested a full tablet report, sending...
I20240502 14:13:46.654407 9390 ts_manager.cc:194] Registered new tserver with Master: 5968a421103245be945b2453edc11c01 (127.6.196.193:33389)
I20240502 14:13:46.655754 9390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.193:37699
W20240502 14:13:46.942129 9583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:46.942430 9583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:46.942567 9583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:46.949442 9583 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:46.949554 9583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:46.949716 9583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.194
I20240502 14:13:46.957727 9583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.196.194
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.194
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:46.958784 9583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:46.960302 9583 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:46.971042 9588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.971757 9589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:46.972411 9591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:46.973153 9583 server_base.cc:1034] running on GCE node
I20240502 14:13:46.973717 9583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:46.974447 9583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:46.975626 9583 hybrid_clock.cc:648] HybridClock initialized: now 1714659226975580 us; error 36 us; skew 500 ppm
I20240502 14:13:46.975884 9583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:46.977725 9583 webserver.cc:460] Webserver started at http://127.6.196.194:41903/ using document root <none> and password file <none>
I20240502 14:13:46.978303 9583 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:46.978431 9583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:46.978695 9583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:46.981112 9583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "39deda1478724559830a8c119eae5be0"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-kjf7"
I20240502 14:13:46.981954 9583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "39deda1478724559830a8c119eae5be0"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-kjf7"
I20240502 14:13:46.986236 9583 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:13:46.988958 9597 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:46.989986 9583 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:13:46.990116 9583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "39deda1478724559830a8c119eae5be0"
format_stamp: "Formatted at 2024-05-02 14:13:46 on dist-test-slave-kjf7"
I20240502 14:13:46.990270 9583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:47.007781 9583 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:47.008406 9583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:47.008649 9583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:47.009505 9583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:47.010807 9583 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:47.010898 9583 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:47.010999 9583 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:47.011072 9583 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:47.039631 9583 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.194:35583
I20240502 14:13:47.039685 9713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.194:35583 every 8 connection(s)
I20240502 14:13:47.040805 9583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:13:47.043762 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 9583
I20240502 14:13:47.043926 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:13:47.046587 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.195:0
--local_ip_for_outbound_sockets=127.6.196.195
--webserver_interface=127.6.196.195
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:47.052320 9714 heartbeater.cc:347] Connected to a master server at 127.6.196.254:43585
I20240502 14:13:47.052594 9714 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:47.053332 9714 heartbeater.cc:511] Master 127.6.196.254:43585 requested a full tablet report, sending...
I20240502 14:13:47.054571 9390 ts_manager.cc:194] Registered new tserver with Master: 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583)
I20240502 14:13:47.055228 9390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.194:58297
W20240502 14:13:47.355041 9717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:47.355334 9717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:47.355474 9717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:47.362736 9717 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:47.362871 9717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:47.363027 9717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.195
I20240502 14:13:47.372005 9717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.196.195
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.195
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:47.373334 9717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:47.375118 9717 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:47.387153 9722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:47.387732 9725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:47.388299 9723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:47.388936 9717 server_base.cc:1034] running on GCE node
I20240502 14:13:47.389524 9717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:47.390385 9717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:47.391583 9717 hybrid_clock.cc:648] HybridClock initialized: now 1714659227391534 us; error 44 us; skew 500 ppm
I20240502 14:13:47.391868 9717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:47.394006 9717 webserver.cc:460] Webserver started at http://127.6.196.195:45911/ using document root <none> and password file <none>
I20240502 14:13:47.394742 9717 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:47.394852 9717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:47.395150 9717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:47.397665 9717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2c6030342d9b40bd91c198e3634a041a"
format_stamp: "Formatted at 2024-05-02 14:13:47 on dist-test-slave-kjf7"
I20240502 14:13:47.398363 9717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2c6030342d9b40bd91c198e3634a041a"
format_stamp: "Formatted at 2024-05-02 14:13:47 on dist-test-slave-kjf7"
I20240502 14:13:47.403064 9717 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 14:13:47.406008 9732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:47.407176 9717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:13:47.407321 9717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2c6030342d9b40bd91c198e3634a041a"
format_stamp: "Formatted at 2024-05-02 14:13:47 on dist-test-slave-kjf7"
I20240502 14:13:47.407485 9717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:47.432277 9717 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:47.433043 9717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:47.433321 9717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:47.434129 9717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:47.435331 9717 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:47.435423 9717 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:47.435523 9717 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:47.435588 9717 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:47.463825 9717 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.195:41811
I20240502 14:13:47.463884 9846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.195:41811 every 8 connection(s)
I20240502 14:13:47.465094 9717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:13:47.473517 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 9717
I20240502 14:13:47.473739 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:13:47.475787 9847 heartbeater.cc:347] Connected to a master server at 127.6.196.254:43585
I20240502 14:13:47.476045 9847 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:47.476608 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.196:0
--local_ip_for_outbound_sockets=127.6.196.196
--webserver_interface=127.6.196.196
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:47.476738 9847 heartbeater.cc:511] Master 127.6.196.254:43585 requested a full tablet report, sending...
I20240502 14:13:47.478078 9390 ts_manager.cc:194] Registered new tserver with Master: 2c6030342d9b40bd91c198e3634a041a (127.6.196.195:41811)
I20240502 14:13:47.479146 9390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.195:42267
I20240502 14:13:47.658244 9578 heartbeater.cc:503] Master 127.6.196.254:43585 was elected leader, sending a full tablet report...
W20240502 14:13:47.767866 9850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:47.768133 9850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:47.768256 9850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:47.774926 9850 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:47.775043 9850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:47.775204 9850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.196
I20240502 14:13:47.783789 9850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.196.196
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.196
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:47.784899 9850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:47.786504 9850 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:47.797868 9856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:47.798424 9858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:47.799175 9855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:47.799970 9850 server_base.cc:1034] running on GCE node
I20240502 14:13:47.800462 9850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:47.801326 9850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:47.802548 9850 hybrid_clock.cc:648] HybridClock initialized: now 1714659227802477 us; error 49 us; skew 500 ppm
I20240502 14:13:47.802816 9850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:47.804822 9850 webserver.cc:460] Webserver started at http://127.6.196.196:37429/ using document root <none> and password file <none>
I20240502 14:13:47.805485 9850 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:47.805575 9850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:47.805826 9850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:47.808118 9850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "33de03b632d64afe8eca2f78b08fa42e"
format_stamp: "Formatted at 2024-05-02 14:13:47 on dist-test-slave-kjf7"
I20240502 14:13:47.808796 9850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "33de03b632d64afe8eca2f78b08fa42e"
format_stamp: "Formatted at 2024-05-02 14:13:47 on dist-test-slave-kjf7"
I20240502 14:13:47.813382 9850 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20240502 14:13:47.816426 9864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:47.817656 9850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:13:47.817780 9850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "33de03b632d64afe8eca2f78b08fa42e"
format_stamp: "Formatted at 2024-05-02 14:13:47 on dist-test-slave-kjf7"
I20240502 14:13:47.817929 9850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:47.839075 9850 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:47.839778 9850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:47.839998 9850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:47.840795 9850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:47.842164 9850 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:47.842269 9850 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:47.842344 9850 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:47.842398 9850 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:47.873023 9850 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.196:32821
I20240502 14:13:47.873064 9980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.196:32821 every 8 connection(s)
I20240502 14:13:47.874363 9850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:13:47.882911 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 9850
I20240502 14:13:47.883088 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:13:47.886488 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.197:0
--local_ip_for_outbound_sockets=127.6.196.197
--webserver_interface=127.6.196.197
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:47.886724 9981 heartbeater.cc:347] Connected to a master server at 127.6.196.254:43585
I20240502 14:13:47.886960 9981 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:47.887595 9981 heartbeater.cc:511] Master 127.6.196.254:43585 requested a full tablet report, sending...
I20240502 14:13:47.888911 9390 ts_manager.cc:194] Registered new tserver with Master: 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196:32821)
I20240502 14:13:47.889961 9390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.196:33403
I20240502 14:13:48.057155 9714 heartbeater.cc:503] Master 127.6.196.254:43585 was elected leader, sending a full tablet report...
W20240502 14:13:48.188922 9984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:48.189188 9984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:48.189342 9984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:48.196146 9984 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:48.196269 9984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:48.196435 9984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.197
I20240502 14:13:48.204712 9984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.196.197
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.197
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:48.205911 9984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:48.207494 9984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:48.218179 9990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:48.218930 9993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:48.219465 9991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:48.220103 9984 server_base.cc:1034] running on GCE node
I20240502 14:13:48.220572 9984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:48.221529 9984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:48.222748 9984 hybrid_clock.cc:648] HybridClock initialized: now 1714659228222682 us; error 55 us; skew 500 ppm
I20240502 14:13:48.223026 9984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:48.225070 9984 webserver.cc:460] Webserver started at http://127.6.196.197:45727/ using document root <none> and password file <none>
I20240502 14:13:48.225726 9984 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:48.225821 9984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:48.226092 9984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:48.228343 9984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "2f7a0e54a60741298e73367e8bbf1237"
format_stamp: "Formatted at 2024-05-02 14:13:48 on dist-test-slave-kjf7"
I20240502 14:13:48.229000 9984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "2f7a0e54a60741298e73367e8bbf1237"
format_stamp: "Formatted at 2024-05-02 14:13:48 on dist-test-slave-kjf7"
I20240502 14:13:48.233620 9984 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240502 14:13:48.236558 9999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:48.237704 9984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:13:48.237834 9984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2f7a0e54a60741298e73367e8bbf1237"
format_stamp: "Formatted at 2024-05-02 14:13:48 on dist-test-slave-kjf7"
I20240502 14:13:48.237996 9984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:48.258178 9984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:48.258822 9984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:48.259049 9984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:48.259855 9984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:48.261197 9984 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:48.261310 9984 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:48.261402 9984 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:48.261473 9984 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:48.290266 9984 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.197:45913
I20240502 14:13:48.290330 10118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.197:45913 every 8 connection(s)
I20240502 14:13:48.291374 9984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:13:48.292112 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 9984
I20240502 14:13:48.292304 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:13:48.295796 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.198:0
--local_ip_for_outbound_sockets=127.6.196.198
--webserver_interface=127.6.196.198
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:48.304306 10119 heartbeater.cc:347] Connected to a master server at 127.6.196.254:43585
I20240502 14:13:48.304564 10119 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:48.305217 10119 heartbeater.cc:511] Master 127.6.196.254:43585 requested a full tablet report, sending...
I20240502 14:13:48.306473 9390 ts_manager.cc:194] Registered new tserver with Master: 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197:45913)
I20240502 14:13:48.307210 9390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.197:42079
I20240502 14:13:48.481439 9847 heartbeater.cc:503] Master 127.6.196.254:43585 was elected leader, sending a full tablet report...
W20240502 14:13:48.594946 10122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:48.595232 10122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:48.595362 10122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:48.602255 10122 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:48.602373 10122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:48.602516 10122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.198
I20240502 14:13:48.610442 10122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.196.198
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.198
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:48.611510 10122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:48.613018 10122 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:48.623401 10127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:48.624037 10130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:48.624749 10128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:48.625556 10122 server_base.cc:1034] running on GCE node
I20240502 14:13:48.626063 10122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:48.626838 10122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:48.628012 10122 hybrid_clock.cc:648] HybridClock initialized: now 1714659228627983 us; error 23 us; skew 500 ppm
I20240502 14:13:48.628270 10122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:48.630097 10122 webserver.cc:460] Webserver started at http://127.6.196.198:41301/ using document root <none> and password file <none>
I20240502 14:13:48.630697 10122 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:48.630805 10122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:48.631085 10122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:48.633282 10122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "99b8f6d3810b4240ba87893565a08ce9"
format_stamp: "Formatted at 2024-05-02 14:13:48 on dist-test-slave-kjf7"
I20240502 14:13:48.633989 10122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "99b8f6d3810b4240ba87893565a08ce9"
format_stamp: "Formatted at 2024-05-02 14:13:48 on dist-test-slave-kjf7"
I20240502 14:13:48.638218 10122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:13:48.641739 10136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:48.642979 10122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:13:48.643132 10122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "99b8f6d3810b4240ba87893565a08ce9"
format_stamp: "Formatted at 2024-05-02 14:13:48 on dist-test-slave-kjf7"
I20240502 14:13:48.643342 10122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:48.667320 10122 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:48.667933 10122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:48.668174 10122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:48.668900 10122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:48.670152 10122 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:48.670240 10122 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:48.670344 10122 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:48.670418 10122 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:48.697321 10122 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.198:34211
I20240502 14:13:48.697388 10250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.198:34211 every 8 connection(s)
I20240502 14:13:48.698421 10122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:13:48.702023 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 10122
I20240502 14:13:48.702159 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:13:48.704829 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.199:0
--local_ip_for_outbound_sockets=127.6.196.199
--webserver_interface=127.6.196.199
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:48.709204 10251 heartbeater.cc:347] Connected to a master server at 127.6.196.254:43585
I20240502 14:13:48.709512 10251 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:48.710359 10251 heartbeater.cc:511] Master 127.6.196.254:43585 requested a full tablet report, sending...
I20240502 14:13:48.711650 9390 ts_manager.cc:194] Registered new tserver with Master: 99b8f6d3810b4240ba87893565a08ce9 (127.6.196.198:34211)
I20240502 14:13:48.712366 9390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.198:53461
I20240502 14:13:48.892215 9981 heartbeater.cc:503] Master 127.6.196.254:43585 was elected leader, sending a full tablet report...
W20240502 14:13:48.994740 10254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:48.994963 10254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:48.995090 10254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:49.001308 10254 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:49.001396 10254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:49.001549 10254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.199
I20240502 14:13:49.009106 10254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.196.199
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.199
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:49.010231 10254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:49.011911 10254 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:49.023159 10259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:49.023834 10262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:49.024474 10260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:49.025168 10254 server_base.cc:1034] running on GCE node
I20240502 14:13:49.025668 10254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:49.026506 10254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:49.027699 10254 hybrid_clock.cc:648] HybridClock initialized: now 1714659229027657 us; error 43 us; skew 500 ppm
I20240502 14:13:49.027977 10254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:49.030162 10254 webserver.cc:460] Webserver started at http://127.6.196.199:36817/ using document root <none> and password file <none>
I20240502 14:13:49.030818 10254 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:49.030920 10254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:49.031188 10254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:13:49.033524 10254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "b4b9d2cad7e04837bc20566099917739"
format_stamp: "Formatted at 2024-05-02 14:13:49 on dist-test-slave-kjf7"
I20240502 14:13:49.034243 10254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "b4b9d2cad7e04837bc20566099917739"
format_stamp: "Formatted at 2024-05-02 14:13:49 on dist-test-slave-kjf7"
I20240502 14:13:49.039017 10254 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:13:49.042266 10268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:49.043639 10254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:13:49.043830 10254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b4b9d2cad7e04837bc20566099917739"
format_stamp: "Formatted at 2024-05-02 14:13:49 on dist-test-slave-kjf7"
I20240502 14:13:49.044040 10254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:49.058162 10254 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:49.058840 10254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:49.059098 10254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:49.060083 10254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:49.061542 10254 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:49.061657 10254 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:49.061774 10254 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:49.061854 10254 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:49.090747 10254 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.199:43261
I20240502 14:13:49.090790 10383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.199:43261 every 8 connection(s)
I20240502 14:13:49.091846 10254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:13:49.100648 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 10254
I20240502 14:13:49.100822 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:13:49.104699 10384 heartbeater.cc:347] Connected to a master server at 127.6.196.254:43585
I20240502 14:13:49.104907 10384 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:49.105607 10384 heartbeater.cc:511] Master 127.6.196.254:43585 requested a full tablet report, sending...
I20240502 14:13:49.106645 9390 ts_manager.cc:194] Registered new tserver with Master: b4b9d2cad7e04837bc20566099917739 (127.6.196.199:43261)
I20240502 14:13:49.107317 9390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.199:49051
I20240502 14:13:49.116245 6931 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 14:13:49.131078 9390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40994:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240502 14:13:49.154784 10185 tablet_service.cc:1467] Processing CreateTablet for tablet 5f8493b9da2c4cf29de9a1ac4c70cac0 (DEFAULT_TABLE table=TestTable [id=9dad9663012f4221a9438181a90cc992]), partition=RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:49.155881 10185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f8493b9da2c4cf29de9a1ac4c70cac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:49.164037 10399 tablet_bootstrap.cc:492] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9: Bootstrap starting.
I20240502 14:13:49.166432 10399 tablet_bootstrap.cc:654] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:49.167388 10399 log.cc:824] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:49.169437 10399 tablet_bootstrap.cc:492] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9: No bootstrap required, opened a new log
I20240502 14:13:49.169615 10399 ts_tablet_manager.cc:1367] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:13:49.174741 10399 raft_consensus.cc:348] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } }
I20240502 14:13:49.174991 10399 raft_consensus.cc:374] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:49.175063 10399 raft_consensus.cc:729] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b8f6d3810b4240ba87893565a08ce9, State: Initialized, Role: FOLLOWER
I20240502 14:13:49.175551 10399 consensus_queue.cc:260] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } }
I20240502 14:13:49.175714 10399 raft_consensus.cc:388] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:49.175801 10399 raft_consensus.cc:482] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:49.175905 10399 raft_consensus.cc:3037] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:49.177513 10399 raft_consensus.cc:504] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } }
I20240502 14:13:49.177861 10399 leader_election.cc:302] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99b8f6d3810b4240ba87893565a08ce9; no voters:
I20240502 14:13:49.178292 10399 leader_election.cc:288] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:49.178424 10401 raft_consensus.cc:2781] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:49.178907 10401 raft_consensus.cc:686] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 LEADER]: Becoming Leader. State: Replica: 99b8f6d3810b4240ba87893565a08ce9, State: Running, Role: LEADER
I20240502 14:13:49.179211 10251 heartbeater.cc:503] Master 127.6.196.254:43585 was elected leader, sending a full tablet report...
I20240502 14:13:49.179417 10401 consensus_queue.cc:237] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } }
I20240502 14:13:49.179662 10399 ts_tablet_manager.cc:1398] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20240502 14:13:49.184185 9390 catalog_manager.cc:5484] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99b8f6d3810b4240ba87893565a08ce9 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "99b8f6d3810b4240ba87893565a08ce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:49.198041 6931 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240502 14:13:49.199679 6931 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5968a421103245be945b2453edc11c01 to finish bootstrapping
I20240502 14:13:49.208228 6931 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 39deda1478724559830a8c119eae5be0 to finish bootstrapping
I20240502 14:13:49.215438 6931 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2c6030342d9b40bd91c198e3634a041a to finish bootstrapping
I20240502 14:13:49.223023 6931 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 33de03b632d64afe8eca2f78b08fa42e to finish bootstrapping
I20240502 14:13:49.229825 6931 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 2f7a0e54a60741298e73367e8bbf1237 to finish bootstrapping
I20240502 14:13:49.242020 6931 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 99b8f6d3810b4240ba87893565a08ce9 to finish bootstrapping
I20240502 14:13:49.251489 6931 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b4b9d2cad7e04837bc20566099917739 to finish bootstrapping
I20240502 14:13:49.259074 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 9717
I20240502 14:13:49.259958 9842 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:49.309298 10119 heartbeater.cc:503] Master 127.6.196.254:43585 was elected leader, sending a full tablet report...
I20240502 14:13:49.331527 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 9717
I20240502 14:13:49.350729 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 9850
I20240502 14:13:49.351686 9976 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:49.422336 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 9850
I20240502 14:13:49.437757 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 9984
I20240502 14:13:49.438680 10114 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:49.511890 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 9984
I20240502 14:13:49.527738 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 10122
I20240502 14:13:49.528856 10246 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:49.607925 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 10122
I20240502 14:13:49.623566 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 10254
I20240502 14:13:49.624471 10379 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:13:49.695883 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 10254
I20240502 14:13:53.462182 9390 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40994:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:13:53.479619 9511 tablet_service.cc:1467] Processing CreateTablet for tablet 63746cf04e6145749291b0c9e28b4b82 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e2fdab079d66498eb09863f56c17fbf2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:53.479590 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 93d3db61bd8143d48d66ce379e443323 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e2fdab079d66498eb09863f56c17fbf2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:53.481317 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63746cf04e6145749291b0c9e28b4b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:53.481707 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93d3db61bd8143d48d66ce379e443323. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:53.481868 9647 tablet_service.cc:1467] Processing CreateTablet for tablet cf42265c4952414d90b9ac9d8fb934e6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=e2fdab079d66498eb09863f56c17fbf2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:53.482825 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf42265c4952414d90b9ac9d8fb934e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:53.491608 10428 tablet_bootstrap.cc:492] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0: Bootstrap starting.
I20240502 14:13:53.493945 10429 tablet_bootstrap.cc:492] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01: Bootstrap starting.
I20240502 14:13:53.494793 10428 tablet_bootstrap.cc:654] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:53.495724 10428 log.cc:824] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:53.496315 10429 tablet_bootstrap.cc:654] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:53.497285 10429 log.cc:824] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:53.498137 10428 tablet_bootstrap.cc:492] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0: No bootstrap required, opened a new log
I20240502 14:13:53.498346 10428 ts_tablet_manager.cc:1367] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20240502 14:13:53.499516 10429 tablet_bootstrap.cc:492] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01: No bootstrap required, opened a new log
I20240502 14:13:53.499707 10429 ts_tablet_manager.cc:1367] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:13:53.503556 10428 raft_consensus.cc:348] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:53.503847 10428 raft_consensus.cc:374] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:53.503937 10428 raft_consensus.cc:729] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39deda1478724559830a8c119eae5be0, State: Initialized, Role: FOLLOWER
I20240502 14:13:53.504505 10428 consensus_queue.cc:260] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:53.504678 10428 raft_consensus.cc:388] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:53.504819 10428 raft_consensus.cc:482] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:53.504985 10428 raft_consensus.cc:3037] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:53.504895 10429 raft_consensus.cc:348] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:53.505203 10429 raft_consensus.cc:374] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:53.505306 10429 raft_consensus.cc:729] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Initialized, Role: FOLLOWER
I20240502 14:13:53.505790 10429 consensus_queue.cc:260] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:53.505954 10429 raft_consensus.cc:388] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:53.506069 10429 raft_consensus.cc:482] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:53.506196 10429 raft_consensus.cc:3037] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:53.506898 10428 raft_consensus.cc:504] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:53.507277 10428 leader_election.cc:302] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39deda1478724559830a8c119eae5be0; no voters:
I20240502 14:13:53.507849 10428 leader_election.cc:288] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:53.507858 10429 raft_consensus.cc:504] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:53.507948 10432 raft_consensus.cc:2781] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:53.508282 10429 leader_election.cc:302] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5968a421103245be945b2453edc11c01; no voters:
I20240502 14:13:53.509528 10428 ts_tablet_manager.cc:1398] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20240502 14:13:53.509966 10432 raft_consensus.cc:686] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Becoming Leader. State: Replica: 39deda1478724559830a8c119eae5be0, State: Running, Role: LEADER
I20240502 14:13:53.510237 10429 leader_election.cc:288] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:53.510365 10435 raft_consensus.cc:2781] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:53.510406 10432 consensus_queue.cc:237] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:53.511703 10429 ts_tablet_manager.cc:1398] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01: Time spent starting tablet: real 0.012s user 0.000s sys 0.012s
I20240502 14:13:53.512058 10429 tablet_bootstrap.cc:492] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01: Bootstrap starting.
I20240502 14:13:53.513593 10429 tablet_bootstrap.cc:654] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:53.515218 9389 catalog_manager.cc:5484] T cf42265c4952414d90b9ac9d8fb934e6 P 39deda1478724559830a8c119eae5be0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39deda1478724559830a8c119eae5be0 (127.6.196.194). New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:53.516465 10429 tablet_bootstrap.cc:492] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01: No bootstrap required, opened a new log
I20240502 14:13:53.516582 10429 ts_tablet_manager.cc:1367] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:13:53.517295 10429 raft_consensus.cc:348] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:53.517449 10429 raft_consensus.cc:374] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:53.517547 10429 raft_consensus.cc:729] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Initialized, Role: FOLLOWER
I20240502 14:13:53.517733 10429 consensus_queue.cc:260] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:53.517889 10429 raft_consensus.cc:388] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:53.517956 10435 raft_consensus.cc:686] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Leader. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:13:53.517985 10429 raft_consensus.cc:482] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:53.518138 10429 raft_consensus.cc:3037] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:53.518421 10435 consensus_queue.cc:237] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:53.520931 10429 raft_consensus.cc:504] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:53.521205 10429 leader_election.cc:302] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5968a421103245be945b2453edc11c01; no voters:
I20240502 14:13:53.521420 10429 leader_election.cc:288] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:53.521530 10437 raft_consensus.cc:2781] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:53.521739 10437 raft_consensus.cc:686] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Leader. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:13:53.521845 10429 ts_tablet_manager.cc:1398] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20240502 14:13:53.522099 10437 consensus_queue.cc:237] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:53.522511 9389 catalog_manager.cc:5484] T 93d3db61bd8143d48d66ce379e443323 P 5968a421103245be945b2453edc11c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5968a421103245be945b2453edc11c01 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:53.526427 9390 catalog_manager.cc:5484] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5968a421103245be945b2453edc11c01 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } health_report { overall_health: HEALTHY } } }
W20240502 14:13:53.840119 10444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:53.840358 10444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:53.846966 10444 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 126.273 ms
time per row: 0.0631366 ms
I20240502 14:13:54.099210 9389 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40994:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:13:54.106567 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 594efe32e19c46658945080d250ca29c (DEFAULT_TABLE table=rebalance_test_table_1 [id=da24f830d1484a7391d1079e2ef8ca05]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:54.106582 9647 tablet_service.cc:1467] Processing CreateTablet for tablet 4ae2df4e083e4205a815a77897710ab1 (DEFAULT_TABLE table=rebalance_test_table_1 [id=da24f830d1484a7391d1079e2ef8ca05]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:54.107018 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae2df4e083e4205a815a77897710ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:54.107082 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 9a2af91313e0451391e05d10dce3c2ea (DEFAULT_TABLE table=rebalance_test_table_1 [id=da24f830d1484a7391d1079e2ef8ca05]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:54.107184 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594efe32e19c46658945080d250ca29c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:54.107373 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2af91313e0451391e05d10dce3c2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:54.111450 10482 tablet_bootstrap.cc:492] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01: Bootstrap starting.
I20240502 14:13:54.113219 10482 tablet_bootstrap.cc:654] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:54.113346 10483 tablet_bootstrap.cc:492] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0: Bootstrap starting.
I20240502 14:13:54.114925 10483 tablet_bootstrap.cc:654] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:54.115191 10482 tablet_bootstrap.cc:492] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01: No bootstrap required, opened a new log
I20240502 14:13:54.115311 10482 ts_tablet_manager.cc:1367] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:13:54.115970 10482 raft_consensus.cc:348] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:54.116163 10482 raft_consensus.cc:374] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:54.116236 10482 raft_consensus.cc:729] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Initialized, Role: FOLLOWER
I20240502 14:13:54.116447 10482 consensus_queue.cc:260] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:54.116595 10482 raft_consensus.cc:388] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:54.116649 10482 raft_consensus.cc:482] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:54.116734 10482 raft_consensus.cc:3037] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:54.118180 10483 tablet_bootstrap.cc:492] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0: No bootstrap required, opened a new log
I20240502 14:13:54.118302 10483 ts_tablet_manager.cc:1367] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240502 14:13:54.118899 10483 raft_consensus.cc:348] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:54.119027 10483 raft_consensus.cc:374] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:54.119103 10483 raft_consensus.cc:729] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39deda1478724559830a8c119eae5be0, State: Initialized, Role: FOLLOWER
I20240502 14:13:54.119117 10482 raft_consensus.cc:504] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:54.119321 10482 leader_election.cc:302] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5968a421103245be945b2453edc11c01; no voters:
I20240502 14:13:54.119302 10483 consensus_queue.cc:260] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:54.119436 10483 raft_consensus.cc:388] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:54.119475 10482 leader_election.cc:288] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:54.119506 10483 raft_consensus.cc:482] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:54.119542 10437 raft_consensus.cc:2781] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:54.119603 10483 raft_consensus.cc:3037] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:54.119979 10482 ts_tablet_manager.cc:1398] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240502 14:13:54.121718 10483 raft_consensus.cc:504] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:54.121917 10483 leader_election.cc:302] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39deda1478724559830a8c119eae5be0; no voters:
I20240502 14:13:54.122085 10483 leader_election.cc:288] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:54.122148 10432 raft_consensus.cc:2781] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:54.124038 10432 raft_consensus.cc:686] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Becoming Leader. State: Replica: 39deda1478724559830a8c119eae5be0, State: Running, Role: LEADER
I20240502 14:13:54.124320 10432 consensus_queue.cc:237] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:54.126463 10437 raft_consensus.cc:686] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Leader. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:13:54.126660 10437 consensus_queue.cc:237] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:54.129776 9389 catalog_manager.cc:5484] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39deda1478724559830a8c119eae5be0 (127.6.196.194). New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:54.130012 10483 ts_tablet_manager.cc:1398] T 594efe32e19c46658945080d250ca29c P 39deda1478724559830a8c119eae5be0: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s
I20240502 14:13:54.130199 10483 tablet_bootstrap.cc:492] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0: Bootstrap starting.
I20240502 14:13:54.131340 10483 tablet_bootstrap.cc:654] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:54.133824 10483 tablet_bootstrap.cc:492] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0: No bootstrap required, opened a new log
I20240502 14:13:54.133967 10483 ts_tablet_manager.cc:1367] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:13:54.134657 10483 raft_consensus.cc:348] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:54.134802 10483 raft_consensus.cc:374] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:54.134873 10483 raft_consensus.cc:729] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39deda1478724559830a8c119eae5be0, State: Initialized, Role: FOLLOWER
I20240502 14:13:54.135066 10483 consensus_queue.cc:260] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:54.135202 10483 raft_consensus.cc:388] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:54.135344 10483 raft_consensus.cc:482] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:54.135442 10483 raft_consensus.cc:3037] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:54.138475 10483 raft_consensus.cc:504] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:54.138679 10483 leader_election.cc:302] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39deda1478724559830a8c119eae5be0; no voters:
I20240502 14:13:54.138911 10487 raft_consensus.cc:2781] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:54.138764 9388 catalog_manager.cc:5484] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5968a421103245be945b2453edc11c01 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:54.139111 10487 raft_consensus.cc:686] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Becoming Leader. State: Replica: 39deda1478724559830a8c119eae5be0, State: Running, Role: LEADER
I20240502 14:13:54.139387 10487 consensus_queue.cc:237] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:54.140075 10483 leader_election.cc:288] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:54.141156 10483 ts_tablet_manager.cc:1398] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0: Time spent starting tablet: real 0.007s user 0.000s sys 0.004s
I20240502 14:13:54.142011 9388 catalog_manager.cc:5484] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39deda1478724559830a8c119eae5be0 (127.6.196.194). New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } health_report { overall_health: HEALTHY } } }
W20240502 14:13:54.434460 10493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:54.434718 10493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:54.440762 10493 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 94.1498 ms
time per row: 0.0470749 ms
I20240502 14:13:54.660562 9388 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40994:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:13:54.667965 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 82e72d3de4e34731b7044de67bf6d635 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1af730f9592947898d4b5e6f2b642194]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:54.668202 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 828f6ad5260a469384cad06dd2986b05 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1af730f9592947898d4b5e6f2b642194]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:54.668385 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e72d3de4e34731b7044de67bf6d635. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:54.668323 9511 tablet_service.cc:1467] Processing CreateTablet for tablet ffe9dcb04b354f61a43df073c78aec59 (DEFAULT_TABLE table=rebalance_test_table_2 [id=1af730f9592947898d4b5e6f2b642194]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:54.668601 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828f6ad5260a469384cad06dd2986b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:54.668628 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe9dcb04b354f61a43df073c78aec59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:54.673117 10535 tablet_bootstrap.cc:492] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0: Bootstrap starting.
I20240502 14:13:54.674688 10536 tablet_bootstrap.cc:492] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01: Bootstrap starting.
I20240502 14:13:54.675050 10535 tablet_bootstrap.cc:654] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:54.676200 10536 tablet_bootstrap.cc:654] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:54.676728 10535 tablet_bootstrap.cc:492] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0: No bootstrap required, opened a new log
I20240502 14:13:54.676849 10535 ts_tablet_manager.cc:1367] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:13:54.677568 10535 raft_consensus.cc:348] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:54.677742 10535 raft_consensus.cc:374] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:54.677835 10535 raft_consensus.cc:729] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39deda1478724559830a8c119eae5be0, State: Initialized, Role: FOLLOWER
I20240502 14:13:54.677945 10536 tablet_bootstrap.cc:492] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01: No bootstrap required, opened a new log
I20240502 14:13:54.678066 10536 ts_tablet_manager.cc:1367] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:13:54.678094 10535 consensus_queue.cc:260] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:54.678292 10535 raft_consensus.cc:388] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:54.678365 10535 raft_consensus.cc:482] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:54.678463 10535 raft_consensus.cc:3037] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:54.678788 10536 raft_consensus.cc:348] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:54.678941 10536 raft_consensus.cc:374] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:54.679019 10536 raft_consensus.cc:729] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Initialized, Role: FOLLOWER
I20240502 14:13:54.679224 10536 consensus_queue.cc:260] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:54.679384 10536 raft_consensus.cc:388] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:54.679450 10536 raft_consensus.cc:482] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:54.679559 10536 raft_consensus.cc:3037] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:54.680931 10535 raft_consensus.cc:504] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:54.681198 10535 leader_election.cc:302] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39deda1478724559830a8c119eae5be0; no voters:
I20240502 14:13:54.681466 10432 raft_consensus.cc:2781] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:54.681598 10432 raft_consensus.cc:686] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Becoming Leader. State: Replica: 39deda1478724559830a8c119eae5be0, State: Running, Role: LEADER
I20240502 14:13:54.681818 10432 consensus_queue.cc:237] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:54.681957 10536 raft_consensus.cc:504] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:54.682152 10536 leader_election.cc:302] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5968a421103245be945b2453edc11c01; no voters:
I20240502 14:13:54.682327 10536 leader_election.cc:288] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:54.682420 10437 raft_consensus.cc:2781] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:54.682552 10535 leader_election.cc:288] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:54.683431 10535 ts_tablet_manager.cc:1398] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:13:54.684173 10536 ts_tablet_manager.cc:1398] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:13:54.684194 9388 catalog_manager.cc:5484] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39deda1478724559830a8c119eae5be0 (127.6.196.194). New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:54.684367 10437 raft_consensus.cc:686] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Leader. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:13:54.684463 10536 tablet_bootstrap.cc:492] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01: Bootstrap starting.
I20240502 14:13:54.684577 10437 consensus_queue.cc:237] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:54.685943 10536 tablet_bootstrap.cc:654] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:54.687353 9382 catalog_manager.cc:5484] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5968a421103245be945b2453edc11c01 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:54.689268 10536 tablet_bootstrap.cc:492] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01: No bootstrap required, opened a new log
I20240502 14:13:54.689407 10536 ts_tablet_manager.cc:1367] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 14:13:54.690076 10536 raft_consensus.cc:348] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:54.690297 10536 raft_consensus.cc:374] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:54.690418 10536 raft_consensus.cc:729] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Initialized, Role: FOLLOWER
I20240502 14:13:54.690666 10536 consensus_queue.cc:260] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:54.690794 10536 raft_consensus.cc:388] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:54.690884 10536 raft_consensus.cc:482] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:54.690969 10536 raft_consensus.cc:3037] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:54.692433 10536 raft_consensus.cc:504] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:54.692601 10536 leader_election.cc:302] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5968a421103245be945b2453edc11c01; no voters:
I20240502 14:13:54.692775 10536 leader_election.cc:288] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:54.692806 10437 raft_consensus.cc:2781] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:54.692991 10437 raft_consensus.cc:686] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Leader. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:13:54.693215 10536 ts_tablet_manager.cc:1398] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:13:54.693274 10437 consensus_queue.cc:237] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:54.695082 9388 catalog_manager.cc:5484] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5968a421103245be945b2453edc11c01 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } health_report { overall_health: HEALTHY } } }
W20240502 14:13:54.986749 10546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:54.987030 10546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:54.993291 10546 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 126.273 ms
time per row: 0.0631366 ms
I20240502 14:13:55.244860 9385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40994:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:13:55.252004 9646 tablet_service.cc:1467] Processing CreateTablet for tablet 40c7c0b7f94b42b392afd30752d26196 (DEFAULT_TABLE table=rebalance_test_table_3 [id=21a0de7bef9c4ac685e4c4a9286573eb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:55.252128 9647 tablet_service.cc:1467] Processing CreateTablet for tablet 6adbf15fbe4f49c5b10a30fd608e058a (DEFAULT_TABLE table=rebalance_test_table_3 [id=21a0de7bef9c4ac685e4c4a9286573eb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:55.252431 9646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c7c0b7f94b42b392afd30752d26196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:55.252378 9511 tablet_service.cc:1467] Processing CreateTablet for tablet cb6cacc3e4564116b13c696813ce0426 (DEFAULT_TABLE table=rebalance_test_table_3 [id=21a0de7bef9c4ac685e4c4a9286573eb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:55.252650 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6adbf15fbe4f49c5b10a30fd608e058a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:55.252775 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6cacc3e4564116b13c696813ce0426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:55.258653 10587 tablet_bootstrap.cc:492] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0: Bootstrap starting.
I20240502 14:13:55.259656 10587 tablet_bootstrap.cc:654] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:55.260941 10588 tablet_bootstrap.cc:492] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01: Bootstrap starting.
I20240502 14:13:55.261245 10587 tablet_bootstrap.cc:492] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0: No bootstrap required, opened a new log
I20240502 14:13:55.261366 10587 ts_tablet_manager.cc:1367] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:13:55.261935 10587 raft_consensus.cc:348] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:55.262051 10587 raft_consensus.cc:374] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:55.262123 10587 raft_consensus.cc:729] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39deda1478724559830a8c119eae5be0, State: Initialized, Role: FOLLOWER
I20240502 14:13:55.262248 10588 tablet_bootstrap.cc:654] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:55.262310 10587 consensus_queue.cc:260] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:55.262456 10587 raft_consensus.cc:388] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:55.262522 10587 raft_consensus.cc:482] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:55.262615 10587 raft_consensus.cc:3037] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:55.263660 10588 tablet_bootstrap.cc:492] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01: No bootstrap required, opened a new log
I20240502 14:13:55.263782 10588 ts_tablet_manager.cc:1367] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:13:55.264219 10587 raft_consensus.cc:504] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:55.264326 10588 raft_consensus.cc:348] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:55.264421 10587 leader_election.cc:302] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39deda1478724559830a8c119eae5be0; no voters:
I20240502 14:13:55.264470 10588 raft_consensus.cc:374] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:55.264551 10588 raft_consensus.cc:729] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Initialized, Role: FOLLOWER
I20240502 14:13:55.264568 10587 leader_election.cc:288] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:55.264624 10432 raft_consensus.cc:2781] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:55.264763 10588 consensus_queue.cc:260] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:55.264933 10588 raft_consensus.cc:388] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:55.265013 10588 raft_consensus.cc:482] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:55.265106 10588 raft_consensus.cc:3037] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:55.265792 10587 ts_tablet_manager.cc:1398] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:13:55.266340 10587 tablet_bootstrap.cc:492] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0: Bootstrap starting.
I20240502 14:13:55.266887 10588 raft_consensus.cc:504] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:55.267081 10588 leader_election.cc:302] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5968a421103245be945b2453edc11c01; no voters:
I20240502 14:13:55.267246 10588 leader_election.cc:288] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:55.267319 10437 raft_consensus.cc:2781] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:55.267865 10588 ts_tablet_manager.cc:1398] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:13:55.268185 10437 raft_consensus.cc:686] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Leader. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:13:55.268416 10432 raft_consensus.cc:686] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Becoming Leader. State: Replica: 39deda1478724559830a8c119eae5be0, State: Running, Role: LEADER
I20240502 14:13:55.268414 10437 consensus_queue.cc:237] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:55.268568 10587 tablet_bootstrap.cc:654] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:55.268631 10432 consensus_queue.cc:237] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:55.270606 9385 catalog_manager.cc:5484] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39deda1478724559830a8c119eae5be0 (127.6.196.194). New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:55.271004 10587 tablet_bootstrap.cc:492] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0: No bootstrap required, opened a new log
I20240502 14:13:55.271137 10587 ts_tablet_manager.cc:1367] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:13:55.271708 10587 raft_consensus.cc:348] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:55.271865 10587 raft_consensus.cc:374] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:55.271941 10587 raft_consensus.cc:729] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39deda1478724559830a8c119eae5be0, State: Initialized, Role: FOLLOWER
I20240502 14:13:55.272130 10587 consensus_queue.cc:260] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:55.272250 10587 raft_consensus.cc:388] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:55.272330 10587 raft_consensus.cc:482] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:55.272406 10587 raft_consensus.cc:3037] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:55.273109 9384 catalog_manager.cc:5484] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5968a421103245be945b2453edc11c01 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:55.274739 10587 raft_consensus.cc:504] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:55.274941 10587 leader_election.cc:302] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39deda1478724559830a8c119eae5be0; no voters:
I20240502 14:13:55.275127 10587 leader_election.cc:288] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:55.275251 10432 raft_consensus.cc:2781] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:55.275489 10432 raft_consensus.cc:686] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Becoming Leader. State: Replica: 39deda1478724559830a8c119eae5be0, State: Running, Role: LEADER
I20240502 14:13:55.275671 10587 ts_tablet_manager.cc:1398] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20240502 14:13:55.275712 10432 consensus_queue.cc:237] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:55.277917 9385 catalog_manager.cc:5484] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39deda1478724559830a8c119eae5be0 (127.6.196.194). New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } health_report { overall_health: HEALTHY } } }
W20240502 14:13:55.570842 10598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:55.571132 10598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:55.577819 10598 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 108.389 ms
time per row: 0.0541946 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:13:56.638099 9385 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40994:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:13:56.648013 9511 tablet_service.cc:1467] Processing CreateTablet for tablet c8aba452497f40ec971f5784d3edbfd1 (DEFAULT_TABLE table=rebalance_test_table_4 [id=31c72864384d44d9afa69f8dcd7ef427]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:56.648411 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8aba452497f40ec971f5784d3edbfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:56.648306 9647 tablet_service.cc:1467] Processing CreateTablet for tablet db4824dd6aa44ad7bd2890238e006975 (DEFAULT_TABLE table=rebalance_test_table_4 [id=31c72864384d44d9afa69f8dcd7ef427]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:56.648690 9647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4824dd6aa44ad7bd2890238e006975. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:56.648629 9512 tablet_service.cc:1467] Processing CreateTablet for tablet 7d4a89cb247c4b8ea952c245ca6f6377 (DEFAULT_TABLE table=rebalance_test_table_4 [id=31c72864384d44d9afa69f8dcd7ef427]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:13:56.648949 9512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d4a89cb247c4b8ea952c245ca6f6377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:13:56.655333 10641 tablet_bootstrap.cc:492] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01: Bootstrap starting.
I20240502 14:13:56.655706 10642 tablet_bootstrap.cc:492] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0: Bootstrap starting.
I20240502 14:13:56.656611 10641 tablet_bootstrap.cc:654] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:56.657017 10642 tablet_bootstrap.cc:654] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:56.658900 10642 tablet_bootstrap.cc:492] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0: No bootstrap required, opened a new log
I20240502 14:13:56.659031 10641 tablet_bootstrap.cc:492] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01: No bootstrap required, opened a new log
I20240502 14:13:56.659024 10642 ts_tablet_manager.cc:1367] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:13:56.659147 10641 ts_tablet_manager.cc:1367] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:13:56.659654 10642 raft_consensus.cc:348] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:56.659854 10642 raft_consensus.cc:374] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:56.659737 10641 raft_consensus.cc:348] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:56.659938 10641 raft_consensus.cc:374] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:56.659937 10642 raft_consensus.cc:729] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39deda1478724559830a8c119eae5be0, State: Initialized, Role: FOLLOWER
I20240502 14:13:56.660024 10641 raft_consensus.cc:729] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Initialized, Role: FOLLOWER
I20240502 14:13:56.660145 10642 consensus_queue.cc:260] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:56.660326 10642 raft_consensus.cc:388] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:56.660284 10641 consensus_queue.cc:260] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:56.660400 10642 raft_consensus.cc:482] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:56.660471 10641 raft_consensus.cc:388] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:56.660509 10642 raft_consensus.cc:3037] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:56.660547 10641 raft_consensus.cc:482] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:56.660699 10641 raft_consensus.cc:3037] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:56.662846 10642 raft_consensus.cc:504] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:56.662895 10641 raft_consensus.cc:504] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:56.663066 10642 leader_election.cc:302] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39deda1478724559830a8c119eae5be0; no voters:
I20240502 14:13:56.663092 10641 leader_election.cc:302] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5968a421103245be945b2453edc11c01; no voters:
I20240502 14:13:56.663530 10641 leader_election.cc:288] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:56.663690 10646 raft_consensus.cc:2781] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:56.664182 10642 leader_election.cc:288] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:56.664296 10645 raft_consensus.cc:2781] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:56.664604 10646 raft_consensus.cc:686] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Leader. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:13:56.664906 10646 consensus_queue.cc:237] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:56.665429 10641 ts_tablet_manager.cc:1398] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:13:56.665747 10641 tablet_bootstrap.cc:492] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01: Bootstrap starting.
I20240502 14:13:56.667554 10641 tablet_bootstrap.cc:654] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01: Neither blocks nor log segments found. Creating new log.
I20240502 14:13:56.670539 10645 raft_consensus.cc:686] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Becoming Leader. State: Replica: 39deda1478724559830a8c119eae5be0, State: Running, Role: LEADER
I20240502 14:13:56.670962 10642 ts_tablet_manager.cc:1398] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20240502 14:13:56.670902 10645 consensus_queue.cc:237] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } }
I20240502 14:13:56.673836 9385 catalog_manager.cc:5484] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5968a421103245be945b2453edc11c01 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:56.677059 9383 catalog_manager.cc:5484] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 39deda1478724559830a8c119eae5be0 (127.6.196.194). New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:56.677752 10641 tablet_bootstrap.cc:492] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01: No bootstrap required, opened a new log
I20240502 14:13:56.677922 10641 ts_tablet_manager.cc:1367] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240502 14:13:56.678758 10641 raft_consensus.cc:348] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:56.678959 10641 raft_consensus.cc:374] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:13:56.679044 10641 raft_consensus.cc:729] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Initialized, Role: FOLLOWER
I20240502 14:13:56.679271 10641 consensus_queue.cc:260] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:56.679448 10641 raft_consensus.cc:388] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:56.679556 10641 raft_consensus.cc:482] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:56.679659 10641 raft_consensus.cc:3037] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:13:56.682991 10641 raft_consensus.cc:504] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:56.683164 10641 leader_election.cc:302] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5968a421103245be945b2453edc11c01; no voters:
I20240502 14:13:56.683383 10641 leader_election.cc:288] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:13:56.683434 10647 raft_consensus.cc:2781] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:13:56.683609 10647 raft_consensus.cc:686] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Leader. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:13:56.683950 10641 ts_tablet_manager.cc:1398] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20240502 14:13:56.683879 10647 consensus_queue.cc:237] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } }
I20240502 14:13:56.686165 9383 catalog_manager.cc:5484] T 7d4a89cb247c4b8ea952c245ca6f6377 P 5968a421103245be945b2453edc11c01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5968a421103245be945b2453edc11c01 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } health_report { overall_health: HEALTHY } } }
W20240502 14:13:56.972654 10657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:56.972887 10657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:56.979171 10657 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 119.469 ms
time per row: 0.0597347 ms
I20240502 14:13:57.223801 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.195:41811
--local_ip_for_outbound_sockets=127.6.196.195
--tserver_master_addrs=127.6.196.254:43585
--webserver_port=45911
--webserver_interface=127.6.196.195
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:13:57.512403 10698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:57.512640 10698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:57.512771 10698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:57.519018 10698 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:57.519129 10698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:57.519253 10698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.195
I20240502 14:13:57.526964 10698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.195:41811
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.196.195
--webserver_port=45911
--tserver_master_addrs=127.6.196.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.195
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:57.528012 10698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:57.529446 10698 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:57.540380 10703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:57.541092 10704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:57.541690 10706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:57.542272 10698 server_base.cc:1034] running on GCE node
I20240502 14:13:57.542728 10698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:57.543510 10698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:57.544728 10698 hybrid_clock.cc:648] HybridClock initialized: now 1714659237544671 us; error 45 us; skew 500 ppm
I20240502 14:13:57.544991 10698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:57.546842 10698 webserver.cc:460] Webserver started at http://127.6.196.195:45911/ using document root <none> and password file <none>
I20240502 14:13:57.547421 10698 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:57.547533 10698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:57.551884 10698 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 14:13:57.554196 10712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:57.555298 10698 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:13:57.555439 10698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2c6030342d9b40bd91c198e3634a041a"
format_stamp: "Formatted at 2024-05-02 14:13:47 on dist-test-slave-kjf7"
I20240502 14:13:57.556147 10698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:57.572633 10698 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:57.573302 10698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:57.573559 10698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:57.574422 10698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:57.575675 10698 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:57.575757 10698 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:57.575878 10698 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:57.575932 10698 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:57.603660 10698 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.195:41811
I20240502 14:13:57.603698 10830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.195:41811 every 8 connection(s)
I20240502 14:13:57.604876 10698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:13:57.608779 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 10698
I20240502 14:13:57.609282 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.196:32821
--local_ip_for_outbound_sockets=127.6.196.196
--tserver_master_addrs=127.6.196.254:43585
--webserver_port=37429
--webserver_interface=127.6.196.196
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:57.615729 10831 heartbeater.cc:347] Connected to a master server at 127.6.196.254:43585
I20240502 14:13:57.615973 10831 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:57.616616 10831 heartbeater.cc:511] Master 127.6.196.254:43585 requested a full tablet report, sending...
I20240502 14:13:57.617981 9382 ts_manager.cc:194] Re-registered known tserver with Master: 2c6030342d9b40bd91c198e3634a041a (127.6.196.195:41811)
I20240502 14:13:57.618752 9382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.195:60489
W20240502 14:13:57.900779 10835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:57.901055 10835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:57.901185 10835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:57.907564 10835 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:57.907686 10835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:57.907833 10835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.196
I20240502 14:13:57.915355 10835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.196:32821
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.196.196
--webserver_port=37429
--tserver_master_addrs=127.6.196.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.196
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:57.916338 10835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:57.917819 10835 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:57.928349 10840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:57.928917 10843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:57.929631 10841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:57.930330 10835 server_base.cc:1034] running on GCE node
I20240502 14:13:57.930838 10835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:57.931612 10835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:57.932802 10835 hybrid_clock.cc:648] HybridClock initialized: now 1714659237932759 us; error 31 us; skew 500 ppm
I20240502 14:13:57.933076 10835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:57.935196 10835 webserver.cc:460] Webserver started at http://127.6.196.196:37429/ using document root <none> and password file <none>
I20240502 14:13:57.935817 10835 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:57.935943 10835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:57.940485 10835 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:13:57.942826 10850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:57.944025 10835 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:13:57.944200 10835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "33de03b632d64afe8eca2f78b08fa42e"
format_stamp: "Formatted at 2024-05-02 14:13:47 on dist-test-slave-kjf7"
I20240502 14:13:57.945008 10835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:57.973292 10835 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:57.974059 10835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:57.974352 10835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:57.975239 10835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:57.976511 10835 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:57.976580 10835 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:57.976666 10835 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:57.976708 10835 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:58.005813 10835 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.196:32821
I20240502 14:13:58.005873 10963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.196:32821 every 8 connection(s)
I20240502 14:13:58.007089 10835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:13:58.015283 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 10835
I20240502 14:13:58.015971 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.197:45913
--local_ip_for_outbound_sockets=127.6.196.197
--tserver_master_addrs=127.6.196.254:43585
--webserver_port=45727
--webserver_interface=127.6.196.197
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:58.017774 10964 heartbeater.cc:347] Connected to a master server at 127.6.196.254:43585
I20240502 14:13:58.018004 10964 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:58.018652 10964 heartbeater.cc:511] Master 127.6.196.254:43585 requested a full tablet report, sending...
I20240502 14:13:58.020542 9382 ts_manager.cc:194] Re-registered known tserver with Master: 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196:32821)
I20240502 14:13:58.021281 9382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.196:37433
W20240502 14:13:58.302019 10967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:58.302261 10967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:58.302340 10967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:58.308715 10967 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:58.308816 10967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:58.308907 10967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.197
I20240502 14:13:58.316440 10967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.197:45913
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.196.197
--webserver_port=45727
--tserver_master_addrs=127.6.196.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.197
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:58.317545 10967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:58.318987 10967 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:58.330061 10973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:58.330560 10974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:58.331492 10976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:58.332213 10967 server_base.cc:1034] running on GCE node
I20240502 14:13:58.332698 10967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:58.333564 10967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:58.334779 10967 hybrid_clock.cc:648] HybridClock initialized: now 1714659238334738 us; error 42 us; skew 500 ppm
I20240502 14:13:58.335036 10967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:58.336990 10967 webserver.cc:460] Webserver started at http://127.6.196.197:45727/ using document root <none> and password file <none>
I20240502 14:13:58.337646 10967 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:58.337769 10967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:58.342139 10967 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:13:58.344543 10982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:58.345628 10967 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:13:58.345762 10967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "2f7a0e54a60741298e73367e8bbf1237"
format_stamp: "Formatted at 2024-05-02 14:13:48 on dist-test-slave-kjf7"
I20240502 14:13:58.346426 10967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:58.365407 10967 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:58.365988 10967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:58.366212 10967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:58.366919 10967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:58.368229 10967 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:58.368302 10967 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:58.368413 10967 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:58.368466 10967 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:58.396212 10967 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.197:45913
I20240502 14:13:58.396245 11095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.197:45913 every 8 connection(s)
I20240502 14:13:58.397418 10967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:13:58.401829 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 10967
I20240502 14:13:58.402442 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.198:34211
--local_ip_for_outbound_sockets=127.6.196.198
--tserver_master_addrs=127.6.196.254:43585
--webserver_port=41301
--webserver_interface=127.6.196.198
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:58.409075 11096 heartbeater.cc:347] Connected to a master server at 127.6.196.254:43585
I20240502 14:13:58.409314 11096 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:58.409911 11096 heartbeater.cc:511] Master 127.6.196.254:43585 requested a full tablet report, sending...
I20240502 14:13:58.411065 9382 ts_manager.cc:194] Re-registered known tserver with Master: 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197:45913)
I20240502 14:13:58.411720 9382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.197:33489
I20240502 14:13:58.621006 10831 heartbeater.cc:503] Master 127.6.196.254:43585 was elected leader, sending a full tablet report...
W20240502 14:13:58.689464 11099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:58.689680 11099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:58.689795 11099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:58.696473 11099 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:58.696578 11099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:58.696861 11099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.198
I20240502 14:13:58.704375 11099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.198:34211
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.196.198
--webserver_port=41301
--tserver_master_addrs=127.6.196.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.198
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:58.705402 11099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:58.706826 11099 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:58.717404 11104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:58.717937 11105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:58.718641 11107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:58.719367 11099 server_base.cc:1034] running on GCE node
I20240502 14:13:58.719823 11099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:58.720654 11099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:58.721825 11099 hybrid_clock.cc:648] HybridClock initialized: now 1714659238721781 us; error 35 us; skew 500 ppm
I20240502 14:13:58.722088 11099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:58.728657 11099 webserver.cc:460] Webserver started at http://127.6.196.198:41301/ using document root <none> and password file <none>
I20240502 14:13:58.729272 11099 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:58.729382 11099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:58.733677 11099 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20240502 14:13:58.736102 11113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:58.737112 11099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:13:58.737304 11099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "99b8f6d3810b4240ba87893565a08ce9"
format_stamp: "Formatted at 2024-05-02 14:13:48 on dist-test-slave-kjf7"
I20240502 14:13:58.738034 11099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:58.753136 11099 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:58.753831 11099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:58.754089 11099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:58.754906 11099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:58.756507 11120 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240502 14:13:58.760919 11099 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240502 14:13:58.761008 11099 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20240502 14:13:58.761111 11099 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240502 14:13:58.763422 11099 ts_tablet_manager.cc:590] Registered 1 tablets
I20240502 14:13:58.763520 11099 ts_tablet_manager.cc:569] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20240502 14:13:58.763801 11120 tablet_bootstrap.cc:492] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9: Bootstrap starting.
I20240502 14:13:58.780903 11120 log.cc:824] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9: Log is configured to *not* fsync() on all Append() calls
I20240502 14:13:58.784679 11120 tablet_bootstrap.cc:492] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:13:58.785342 11120 tablet_bootstrap.cc:492] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9: Bootstrap complete.
I20240502 14:13:58.786572 11120 ts_tablet_manager.cc:1367] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.011s
I20240502 14:13:58.795756 11120 raft_consensus.cc:348] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } }
I20240502 14:13:58.796391 11099 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.198:34211
I20240502 14:13:58.796432 11231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.198:34211 every 8 connection(s)
I20240502 14:13:58.796455 11120 raft_consensus.cc:729] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99b8f6d3810b4240ba87893565a08ce9, State: Initialized, Role: FOLLOWER
I20240502 14:13:58.797112 11120 consensus_queue.cc:260] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } }
I20240502 14:13:58.797305 11120 raft_consensus.cc:388] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:13:58.797417 11120 raft_consensus.cc:482] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:13:58.797561 11120 raft_consensus.cc:3037] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:13:58.797631 11099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:13:58.798260 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 11099
I20240502 14:13:58.799115 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.199:43261
--local_ip_for_outbound_sockets=127.6.196.199
--tserver_master_addrs=127.6.196.254:43585
--webserver_port=36817
--webserver_interface=127.6.196.199
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:13:58.802700 11120 raft_consensus.cc:504] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } }
I20240502 14:13:58.803308 11120 leader_election.cc:302] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99b8f6d3810b4240ba87893565a08ce9; no voters:
I20240502 14:13:58.806764 11120 leader_election.cc:288] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240502 14:13:58.808239 11235 raft_consensus.cc:2781] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:13:58.808638 11235 raft_consensus.cc:686] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [term 2 LEADER]: Becoming Leader. State: Replica: 99b8f6d3810b4240ba87893565a08ce9, State: Running, Role: LEADER
I20240502 14:13:58.809080 11235 consensus_queue.cc:237] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } }
I20240502 14:13:58.810281 11120 ts_tablet_manager.cc:1398] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9: Time spent starting tablet: real 0.024s user 0.017s sys 0.005s
I20240502 14:13:58.813391 11232 heartbeater.cc:347] Connected to a master server at 127.6.196.254:43585
I20240502 14:13:58.813707 11232 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:58.814723 11232 heartbeater.cc:511] Master 127.6.196.254:43585 requested a full tablet report, sending...
I20240502 14:13:58.817274 9382 ts_manager.cc:194] Re-registered known tserver with Master: 99b8f6d3810b4240ba87893565a08ce9 (127.6.196.198:34211)
I20240502 14:13:58.817636 9382 catalog_manager.cc:5484] T 5f8493b9da2c4cf29de9a1ac4c70cac0 P 99b8f6d3810b4240ba87893565a08ce9 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "99b8f6d3810b4240ba87893565a08ce9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } health_report { overall_health: HEALTHY } } }
I20240502 14:13:58.821980 9382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.198:46185
I20240502 14:13:58.824355 11232 heartbeater.cc:503] Master 127.6.196.254:43585 was elected leader, sending a full tablet report...
I20240502 14:13:59.023331 10964 heartbeater.cc:503] Master 127.6.196.254:43585 was elected leader, sending a full tablet report...
W20240502 14:13:59.091224 11236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:13:59.091449 11236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:13:59.091568 11236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:13:59.100930 11236 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:13:59.101089 11236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:13:59.101253 11236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.199
I20240502 14:13:59.109608 11236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.199:43261
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.196.199
--webserver_port=36817
--tserver_master_addrs=127.6.196.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.199
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:13:59.110548 11236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:13:59.111936 11236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:13:59.123807 11249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:59.124540 11250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:13:59.125066 11252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:13:59.125877 11236 server_base.cc:1034] running on GCE node
I20240502 14:13:59.126401 11236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:13:59.127247 11236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:13:59.128444 11236 hybrid_clock.cc:648] HybridClock initialized: now 1714659239128391 us; error 35 us; skew 500 ppm
I20240502 14:13:59.128701 11236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:13:59.130596 11236 webserver.cc:460] Webserver started at http://127.6.196.199:36817/ using document root <none> and password file <none>
I20240502 14:13:59.131176 11236 fs_manager.cc:359] Metadata directory not provided
I20240502 14:13:59.131285 11236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:13:59.135777 11236 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:13:59.138128 11258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:59.139356 11236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:13:59.139523 11236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "b4b9d2cad7e04837bc20566099917739"
format_stamp: "Formatted at 2024-05-02 14:13:49 on dist-test-slave-kjf7"
I20240502 14:13:59.140359 11236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:13:59.158912 11236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:13:59.159569 11236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:13:59.159812 11236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:13:59.160578 11236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:13:59.161837 11236 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:13:59.161924 11236 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:59.162060 11236 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:13:59.162110 11236 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:13:59.191085 11236 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.199:43261
I20240502 14:13:59.191129 11373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.199:43261 every 8 connection(s)
I20240502 14:13:59.192283 11236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:13:59.193097 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 11236
I20240502 14:13:59.204927 11374 heartbeater.cc:347] Connected to a master server at 127.6.196.254:43585
I20240502 14:13:59.205155 11374 heartbeater.cc:464] Registering TS with master...
I20240502 14:13:59.205900 11374 heartbeater.cc:511] Master 127.6.196.254:43585 requested a full tablet report, sending...
I20240502 14:13:59.207118 9382 ts_manager.cc:194] Re-registered known tserver with Master: b4b9d2cad7e04837bc20566099917739 (127.6.196.199:43261)
I20240502 14:13:59.207782 9382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.199:40633
I20240502 14:13:59.413812 11096 heartbeater.cc:503] Master 127.6.196.254:43585 was elected leader, sending a full tablet report...
I20240502 14:13:59.603207 11030 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:59.604213 10762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:59.632879 10898 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:59.656361 9511 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:13:59.667084 9647 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:13:59.686561 11308 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:59.699196 11163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:59.840766 10762 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:59.846783 11030 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:59.858672 10898 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:59.877723 9511 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:13:59.879050 9647 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:13:59.903465 11308 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:13:59.908876 11163 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:13:59.970099 9668 consensus_queue.cc:237] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: true } }
I20240502 14:13:59.972219 11463 raft_consensus.cc:2932] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c6030342d9b40bd91c198e3634a041a (127.6.196.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: true } } }
I20240502 14:13:59.978000 9381 catalog_manager.cc:5484] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 reported cstate change: config changed from index -1 to 4, NON_VOTER 2c6030342d9b40bd91c198e3634a041a (127.6.196.195) added. New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:13:59.980926 9598 consensus_peers.cc:485] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 -> Peer 2c6030342d9b40bd91c198e3634a041a (127.6.196.195:41811): Couldn't send request to peer 2c6030342d9b40bd91c198e3634a041a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6adbf15fbe4f49c5b10a30fd608e058a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:13:59.988487 9532 consensus_queue.cc:237] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: true } }
I20240502 14:13:59.990655 11476 raft_consensus.cc:2932] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: true } } }
I20240502 14:13:59.995868 9381 catalog_manager.cc:5484] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 reported cstate change: config changed from index -1 to 4, NON_VOTER 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196) added. New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:13:59.998944 9467 consensus_peers.cc:485] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 -> Peer 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196:32821): Couldn't send request to peer 33de03b632d64afe8eca2f78b08fa42e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82e72d3de4e34731b7044de67bf6d635. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:00.004545 9532 consensus_queue.cc:237] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: true } }
I20240502 14:14:00.006209 11476 raft_consensus.cc:2932] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b4b9d2cad7e04837bc20566099917739 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: true } } }
I20240502 14:14:00.010681 9381 catalog_manager.cc:5484] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 reported cstate change: config changed from index -1 to 4, NON_VOTER b4b9d2cad7e04837bc20566099917739 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:00.014252 9468 consensus_peers.cc:485] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 -> Peer b4b9d2cad7e04837bc20566099917739 (127.6.196.199:43261): Couldn't send request to peer b4b9d2cad7e04837bc20566099917739. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8aba452497f40ec971f5784d3edbfd1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:00.020956 9668 consensus_queue.cc:237] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: true } }
I20240502 14:14:00.022588 11463 raft_consensus.cc:2932] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: true } } }
I20240502 14:14:00.026726 9382 catalog_manager.cc:5484] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 reported cstate change: config changed from index -1 to 4, NON_VOTER 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:00.030256 9601 consensus_peers.cc:485] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 -> Peer 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197:45913): Couldn't send request to peer 2f7a0e54a60741298e73367e8bbf1237. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4ae2df4e083e4205a815a77897710ab1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:00.036940 9532 consensus_queue.cc:237] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: true } }
I20240502 14:14:00.038347 11475 raft_consensus.cc:2932] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 99b8f6d3810b4240ba87893565a08ce9 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: true } } }
I20240502 14:14:00.044618 9382 catalog_manager.cc:5484] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 reported cstate change: config changed from index -1 to 4, NON_VOTER 99b8f6d3810b4240ba87893565a08ce9 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:00.047313 9466 consensus_peers.cc:485] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 -> Peer 99b8f6d3810b4240ba87893565a08ce9 (127.6.196.198:34211): Couldn't send request to peer 99b8f6d3810b4240ba87893565a08ce9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ffe9dcb04b354f61a43df073c78aec59. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:00.056020 9532 consensus_queue.cc:237] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: true } }
I20240502 14:14:00.057416 11475 raft_consensus.cc:2932] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2c6030342d9b40bd91c198e3634a041a (127.6.196.195) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: true } } }
I20240502 14:14:00.061519 9382 catalog_manager.cc:5484] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 reported cstate change: config changed from index -1 to 4, NON_VOTER 2c6030342d9b40bd91c198e3634a041a (127.6.196.195) added. New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:00.062732 9465 consensus_peers.cc:485] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 -> Peer 2c6030342d9b40bd91c198e3634a041a (127.6.196.195:41811): Couldn't send request to peer 2c6030342d9b40bd91c198e3634a041a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a2af91313e0451391e05d10dce3c2ea. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:00.071188 9668 consensus_queue.cc:237] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: true } }
I20240502 14:14:00.072465 11461 raft_consensus.cc:2932] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: true } } }
I20240502 14:14:00.077363 9381 catalog_manager.cc:5484] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 reported cstate change: config changed from index -1 to 4, NON_VOTER 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196) added. New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:00.079324 9600 consensus_peers.cc:485] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 -> Peer 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196:32821): Couldn't send request to peer 33de03b632d64afe8eca2f78b08fa42e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40c7c0b7f94b42b392afd30752d26196. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:00.086010 9532 consensus_queue.cc:237] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: true } }
I20240502 14:14:00.087222 11475 raft_consensus.cc:2932] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER b4b9d2cad7e04837bc20566099917739 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: true } } }
W20240502 14:14:00.087702 9468 consensus_peers.cc:485] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 -> Peer b4b9d2cad7e04837bc20566099917739 (127.6.196.199:43261): Couldn't send request to peer b4b9d2cad7e04837bc20566099917739. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63746cf04e6145749291b0c9e28b4b82. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:00.090724 9381 catalog_manager.cc:5484] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 reported cstate change: config changed from index -1 to 4, NON_VOTER b4b9d2cad7e04837bc20566099917739 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:00.098852 9668 consensus_queue.cc:237] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: true } }
I20240502 14:14:00.100029 11461 raft_consensus.cc:2932] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: true } } }
W20240502 14:14:00.100576 9601 consensus_peers.cc:485] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 -> Peer 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197:45913): Couldn't send request to peer 2f7a0e54a60741298e73367e8bbf1237. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db4824dd6aa44ad7bd2890238e006975. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:00.103817 9382 catalog_manager.cc:5484] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 reported cstate change: config changed from index -1 to 4, NON_VOTER 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:00.209825 11374 heartbeater.cc:503] Master 127.6.196.254:43585 was elected leader, sending a full tablet report...
I20240502 14:14:00.368134 11566 ts_tablet_manager.cc:902] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e: Initiating tablet copy from peer 5968a421103245be945b2453edc11c01 (127.6.196.193:33389)
I20240502 14:14:00.369890 11566 tablet_copy_client.cc:308] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:33389
I20240502 14:14:00.376446 9553 tablet_copy_service.cc:140] P 5968a421103245be945b2453edc11c01: Received BeginTabletCopySession request for tablet 82e72d3de4e34731b7044de67bf6d635 from peer 33de03b632d64afe8eca2f78b08fa42e ({username='slave'} at 127.6.196.196:57787)
I20240502 14:14:00.376696 9553 tablet_copy_service.cc:161] P 5968a421103245be945b2453edc11c01: Beginning new tablet copy session on tablet 82e72d3de4e34731b7044de67bf6d635 from peer 33de03b632d64afe8eca2f78b08fa42e at {username='slave'} at 127.6.196.196:57787: session id = 33de03b632d64afe8eca2f78b08fa42e-82e72d3de4e34731b7044de67bf6d635
I20240502 14:14:00.379370 9553 tablet_copy_source_session.cc:205] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:00.384888 11566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82e72d3de4e34731b7044de67bf6d635. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:00.399586 11566 tablet_copy_client.cc:783] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:00.400004 11566 tablet_copy_client.cc:647] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:00.402453 11566 tablet_copy_client.cc:523] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:00.407064 11566 tablet_bootstrap.cc:492] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e: Bootstrap starting.
I20240502 14:14:00.413805 11593 ts_tablet_manager.cc:902] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739: Initiating tablet copy from peer 5968a421103245be945b2453edc11c01 (127.6.196.193:33389)
I20240502 14:14:00.420644 11566 log.cc:824] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:00.439160 11593 tablet_copy_client.cc:308] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:33389
I20240502 14:14:00.458106 9553 tablet_copy_service.cc:140] P 5968a421103245be945b2453edc11c01: Received BeginTabletCopySession request for tablet c8aba452497f40ec971f5784d3edbfd1 from peer b4b9d2cad7e04837bc20566099917739 ({username='slave'} at 127.6.196.199:39887)
I20240502 14:14:00.458247 9553 tablet_copy_service.cc:161] P 5968a421103245be945b2453edc11c01: Beginning new tablet copy session on tablet c8aba452497f40ec971f5784d3edbfd1 from peer b4b9d2cad7e04837bc20566099917739 at {username='slave'} at 127.6.196.199:39887: session id = b4b9d2cad7e04837bc20566099917739-c8aba452497f40ec971f5784d3edbfd1
I20240502 14:14:00.460775 9553 tablet_copy_source_session.cc:205] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:00.465003 11593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8aba452497f40ec971f5784d3edbfd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:00.475176 11566 tablet_bootstrap.cc:492] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:00.475680 11566 tablet_bootstrap.cc:492] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e: Bootstrap complete.
I20240502 14:14:00.476163 11566 ts_tablet_manager.cc:1367] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e: Time spent bootstrapping tablet: real 0.069s user 0.048s sys 0.020s
I20240502 14:14:00.477283 11593 tablet_copy_client.cc:783] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:00.477589 11593 tablet_copy_client.cc:647] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:00.479840 11593 tablet_copy_client.cc:523] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:00.481421 11566 raft_consensus.cc:348] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: true } }
I20240502 14:14:00.481832 11566 raft_consensus.cc:729] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 33de03b632d64afe8eca2f78b08fa42e, State: Initialized, Role: LEARNER
I20240502 14:14:00.482388 11566 consensus_queue.cc:260] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: true } }
I20240502 14:14:00.483896 11593 tablet_bootstrap.cc:492] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739: Bootstrap starting.
I20240502 14:14:00.495903 11623 ts_tablet_manager.cc:902] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a: Initiating tablet copy from peer 5968a421103245be945b2453edc11c01 (127.6.196.193:33389)
I20240502 14:14:00.498531 11593 log.cc:824] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:00.502647 11566 ts_tablet_manager.cc:1398] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e: Time spent starting tablet: real 0.026s user 0.007s sys 0.004s
I20240502 14:14:00.506177 9553 tablet_copy_service.cc:342] P 5968a421103245be945b2453edc11c01: Request end of tablet copy session 33de03b632d64afe8eca2f78b08fa42e-82e72d3de4e34731b7044de67bf6d635 received from {username='slave'} at 127.6.196.196:57787
I20240502 14:14:00.506327 9553 tablet_copy_service.cc:433] P 5968a421103245be945b2453edc11c01: ending tablet copy session 33de03b632d64afe8eca2f78b08fa42e-82e72d3de4e34731b7044de67bf6d635 on tablet 82e72d3de4e34731b7044de67bf6d635 with peer 33de03b632d64afe8eca2f78b08fa42e
I20240502 14:14:00.507362 11624 ts_tablet_manager.cc:902] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a: Initiating tablet copy from peer 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583)
I20240502 14:14:00.509358 11622 ts_tablet_manager.cc:902] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237: Initiating tablet copy from peer 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583)
I20240502 14:14:00.525710 11631 ts_tablet_manager.cc:902] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9: Initiating tablet copy from peer 5968a421103245be945b2453edc11c01 (127.6.196.193:33389)
I20240502 14:14:00.531854 11633 ts_tablet_manager.cc:902] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739: Initiating tablet copy from peer 5968a421103245be945b2453edc11c01 (127.6.196.193:33389)
I20240502 14:14:00.550164 11643 ts_tablet_manager.cc:902] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237: Initiating tablet copy from peer 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583)
I20240502 14:14:00.551290 11631 tablet_copy_client.cc:308] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:33389
I20240502 14:14:00.568585 11623 tablet_copy_client.cc:308] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:33389
I20240502 14:14:00.581604 11643 tablet_copy_client.cc:308] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:35583
I20240502 14:14:00.591166 9553 tablet_copy_service.cc:140] P 5968a421103245be945b2453edc11c01: Received BeginTabletCopySession request for tablet ffe9dcb04b354f61a43df073c78aec59 from peer 99b8f6d3810b4240ba87893565a08ce9 ({username='slave'} at 127.6.196.198:41159)
I20240502 14:14:00.591367 9553 tablet_copy_service.cc:161] P 5968a421103245be945b2453edc11c01: Beginning new tablet copy session on tablet ffe9dcb04b354f61a43df073c78aec59 from peer 99b8f6d3810b4240ba87893565a08ce9 at {username='slave'} at 127.6.196.198:41159: session id = 99b8f6d3810b4240ba87893565a08ce9-ffe9dcb04b354f61a43df073c78aec59
I20240502 14:14:00.595688 9553 tablet_copy_source_session.cc:205] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:00.595924 11633 tablet_copy_client.cc:308] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:33389
I20240502 14:14:00.596588 9553 tablet_copy_service.cc:140] P 5968a421103245be945b2453edc11c01: Received BeginTabletCopySession request for tablet 63746cf04e6145749291b0c9e28b4b82 from peer b4b9d2cad7e04837bc20566099917739 ({username='slave'} at 127.6.196.199:39887)
I20240502 14:14:00.596791 9553 tablet_copy_service.cc:161] P 5968a421103245be945b2453edc11c01: Beginning new tablet copy session on tablet 63746cf04e6145749291b0c9e28b4b82 from peer b4b9d2cad7e04837bc20566099917739 at {username='slave'} at 127.6.196.199:39887: session id = b4b9d2cad7e04837bc20566099917739-63746cf04e6145749291b0c9e28b4b82
I20240502 14:14:00.600507 9553 tablet_copy_source_session.cc:205] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:00.603775 11631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffe9dcb04b354f61a43df073c78aec59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:00.607367 9689 tablet_copy_service.cc:140] P 39deda1478724559830a8c119eae5be0: Received BeginTabletCopySession request for tablet 4ae2df4e083e4205a815a77897710ab1 from peer 2f7a0e54a60741298e73367e8bbf1237 ({username='slave'} at 127.6.196.197:41027)
I20240502 14:14:00.607534 9689 tablet_copy_service.cc:161] P 39deda1478724559830a8c119eae5be0: Beginning new tablet copy session on tablet 4ae2df4e083e4205a815a77897710ab1 from peer 2f7a0e54a60741298e73367e8bbf1237 at {username='slave'} at 127.6.196.197:41027: session id = 2f7a0e54a60741298e73367e8bbf1237-4ae2df4e083e4205a815a77897710ab1
I20240502 14:14:00.609917 11633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63746cf04e6145749291b0c9e28b4b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:00.610883 9689 tablet_copy_source_session.cc:205] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:00.615499 11643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ae2df4e083e4205a815a77897710ab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:00.620785 9553 tablet_copy_service.cc:140] P 5968a421103245be945b2453edc11c01: Received BeginTabletCopySession request for tablet 9a2af91313e0451391e05d10dce3c2ea from peer 2c6030342d9b40bd91c198e3634a041a ({username='slave'} at 127.6.196.195:43031)
I20240502 14:14:00.621062 9553 tablet_copy_service.cc:161] P 5968a421103245be945b2453edc11c01: Beginning new tablet copy session on tablet 9a2af91313e0451391e05d10dce3c2ea from peer 2c6030342d9b40bd91c198e3634a041a at {username='slave'} at 127.6.196.195:43031: session id = 2c6030342d9b40bd91c198e3634a041a-9a2af91313e0451391e05d10dce3c2ea
I20240502 14:14:00.632019 11566 ts_tablet_manager.cc:902] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e: Initiating tablet copy from peer 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583)
I20240502 14:14:00.632401 9553 tablet_copy_source_session.cc:205] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:00.637478 11623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a2af91313e0451391e05d10dce3c2ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:00.633212 11593 tablet_bootstrap.cc:492] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:00.645692 11593 tablet_bootstrap.cc:492] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739: Bootstrap complete.
I20240502 14:14:00.646081 11593 ts_tablet_manager.cc:1367] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739: Time spent bootstrapping tablet: real 0.163s user 0.082s sys 0.015s
I20240502 14:14:00.648627 11643 tablet_copy_client.cc:783] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:00.648957 11643 tablet_copy_client.cc:647] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:00.650030 11633 tablet_copy_client.cc:783] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:00.650574 11633 tablet_copy_client.cc:647] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:00.651616 11643 tablet_copy_client.cc:523] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:00.654218 11633 tablet_copy_client.cc:523] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:00.657613 11631 tablet_copy_client.cc:783] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:00.659189 11593 raft_consensus.cc:348] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: true } }
I20240502 14:14:00.659600 11593 raft_consensus.cc:729] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4b9d2cad7e04837bc20566099917739, State: Initialized, Role: LEARNER
I20240502 14:14:00.659978 11631 tablet_copy_client.cc:647] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:00.659977 11622 tablet_copy_client.cc:308] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:35583
I20240502 14:14:00.660193 11593 consensus_queue.cc:260] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: true } }
I20240502 14:14:00.660650 9689 tablet_copy_service.cc:140] P 39deda1478724559830a8c119eae5be0: Received BeginTabletCopySession request for tablet db4824dd6aa44ad7bd2890238e006975 from peer 2f7a0e54a60741298e73367e8bbf1237 ({username='slave'} at 127.6.196.197:41027)
I20240502 14:14:00.660830 9689 tablet_copy_service.cc:161] P 39deda1478724559830a8c119eae5be0: Beginning new tablet copy session on tablet db4824dd6aa44ad7bd2890238e006975 from peer 2f7a0e54a60741298e73367e8bbf1237 at {username='slave'} at 127.6.196.197:41027: session id = 2f7a0e54a60741298e73367e8bbf1237-db4824dd6aa44ad7bd2890238e006975
I20240502 14:14:00.662251 11631 tablet_copy_client.cc:523] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:00.663439 9689 tablet_copy_source_session.cc:205] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:00.664566 11622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db4824dd6aa44ad7bd2890238e006975. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:00.664897 11631 tablet_bootstrap.cc:492] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9: Bootstrap starting.
I20240502 14:14:00.666224 11643 tablet_bootstrap.cc:492] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237: Bootstrap starting.
I20240502 14:14:00.669265 11624 tablet_copy_client.cc:308] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:35583
I20240502 14:14:00.670398 11622 tablet_copy_client.cc:783] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:00.670622 11622 tablet_copy_client.cc:647] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:00.672685 11622 tablet_copy_client.cc:523] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:00.674261 11566 tablet_copy_client.cc:308] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:35583
I20240502 14:14:00.675679 11622 tablet_bootstrap.cc:492] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237: Bootstrap starting.
I20240502 14:14:00.679567 9689 tablet_copy_service.cc:140] P 39deda1478724559830a8c119eae5be0: Received BeginTabletCopySession request for tablet 6adbf15fbe4f49c5b10a30fd608e058a from peer 2c6030342d9b40bd91c198e3634a041a ({username='slave'} at 127.6.196.195:60331)
I20240502 14:14:00.679756 9689 tablet_copy_service.cc:161] P 39deda1478724559830a8c119eae5be0: Beginning new tablet copy session on tablet 6adbf15fbe4f49c5b10a30fd608e058a from peer 2c6030342d9b40bd91c198e3634a041a at {username='slave'} at 127.6.196.195:60331: session id = 2c6030342d9b40bd91c198e3634a041a-6adbf15fbe4f49c5b10a30fd608e058a
I20240502 14:14:00.681820 11623 tablet_copy_client.cc:783] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:00.682088 11623 tablet_copy_client.cc:647] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:00.683357 9689 tablet_copy_source_session.cc:205] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:00.683869 11623 tablet_copy_client.cc:523] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:00.685122 11624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6adbf15fbe4f49c5b10a30fd608e058a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:00.686043 11622 log.cc:824] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:00.688118 11623 tablet_bootstrap.cc:492] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a: Bootstrap starting.
I20240502 14:14:00.693094 11633 tablet_bootstrap.cc:492] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739: Bootstrap starting.
I20240502 14:14:00.696650 11624 tablet_copy_client.cc:783] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:00.696835 11624 tablet_copy_client.cc:647] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:00.698912 11624 tablet_copy_client.cc:523] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:00.701071 9689 tablet_copy_service.cc:140] P 39deda1478724559830a8c119eae5be0: Received BeginTabletCopySession request for tablet 40c7c0b7f94b42b392afd30752d26196 from peer 33de03b632d64afe8eca2f78b08fa42e ({username='slave'} at 127.6.196.196:53533)
I20240502 14:14:00.701206 9689 tablet_copy_service.cc:161] P 39deda1478724559830a8c119eae5be0: Beginning new tablet copy session on tablet 40c7c0b7f94b42b392afd30752d26196 from peer 33de03b632d64afe8eca2f78b08fa42e at {username='slave'} at 127.6.196.196:53533: session id = 33de03b632d64afe8eca2f78b08fa42e-40c7c0b7f94b42b392afd30752d26196
I20240502 14:14:00.702894 11624 tablet_bootstrap.cc:492] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a: Bootstrap starting.
I20240502 14:14:00.703915 9689 tablet_copy_source_session.cc:205] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:00.705096 11566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40c7c0b7f94b42b392afd30752d26196. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:00.705942 11593 ts_tablet_manager.cc:1398] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739: Time spent starting tablet: real 0.060s user 0.005s sys 0.005s
I20240502 14:14:00.710875 11566 tablet_copy_client.cc:783] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:00.710947 9553 tablet_copy_service.cc:342] P 5968a421103245be945b2453edc11c01: Request end of tablet copy session b4b9d2cad7e04837bc20566099917739-c8aba452497f40ec971f5784d3edbfd1 received from {username='slave'} at 127.6.196.199:39887
I20240502 14:14:00.711105 11566 tablet_copy_client.cc:647] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:00.711123 9553 tablet_copy_service.cc:433] P 5968a421103245be945b2453edc11c01: ending tablet copy session b4b9d2cad7e04837bc20566099917739-c8aba452497f40ec971f5784d3edbfd1 on tablet c8aba452497f40ec971f5784d3edbfd1 with peer b4b9d2cad7e04837bc20566099917739
I20240502 14:14:00.712765 11566 tablet_copy_client.cc:523] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:00.716193 11566 tablet_bootstrap.cc:492] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e: Bootstrap starting.
I20240502 14:14:00.729547 11623 log.cc:824] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:00.765323 11631 tablet_bootstrap.cc:492] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:00.765769 11631 tablet_bootstrap.cc:492] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9: Bootstrap complete.
I20240502 14:14:00.766076 11631 ts_tablet_manager.cc:1367] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9: Time spent bootstrapping tablet: real 0.101s user 0.048s sys 0.008s
I20240502 14:14:00.767122 11631 raft_consensus.cc:348] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: true } }
I20240502 14:14:00.767366 11631 raft_consensus.cc:729] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 99b8f6d3810b4240ba87893565a08ce9, State: Initialized, Role: LEARNER
I20240502 14:14:00.767592 11631 consensus_queue.cc:260] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: true } }
I20240502 14:14:00.774549 11631 ts_tablet_manager.cc:1398] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9: Time spent starting tablet: real 0.008s user 0.005s sys 0.004s
I20240502 14:14:00.779242 9553 tablet_copy_service.cc:342] P 5968a421103245be945b2453edc11c01: Request end of tablet copy session 99b8f6d3810b4240ba87893565a08ce9-ffe9dcb04b354f61a43df073c78aec59 received from {username='slave'} at 127.6.196.198:41159
I20240502 14:14:00.779430 9553 tablet_copy_service.cc:433] P 5968a421103245be945b2453edc11c01: ending tablet copy session 99b8f6d3810b4240ba87893565a08ce9-ffe9dcb04b354f61a43df073c78aec59 on tablet ffe9dcb04b354f61a43df073c78aec59 with peer 99b8f6d3810b4240ba87893565a08ce9
I20240502 14:14:00.791877 11622 tablet_bootstrap.cc:492] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:00.792557 11622 tablet_bootstrap.cc:492] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237: Bootstrap complete.
I20240502 14:14:00.792963 11622 ts_tablet_manager.cc:1367] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237: Time spent bootstrapping tablet: real 0.117s user 0.059s sys 0.021s
I20240502 14:14:00.802153 11624 tablet_bootstrap.cc:492] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:00.802886 11624 tablet_bootstrap.cc:492] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a: Bootstrap complete.
I20240502 14:14:00.803458 11624 ts_tablet_manager.cc:1367] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a: Time spent bootstrapping tablet: real 0.101s user 0.048s sys 0.009s
I20240502 14:14:00.805509 11623 tablet_bootstrap.cc:492] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:00.805996 11623 tablet_bootstrap.cc:492] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a: Bootstrap complete.
I20240502 14:14:00.806242 11623 ts_tablet_manager.cc:1367] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a: Time spent bootstrapping tablet: real 0.118s user 0.051s sys 0.006s
I20240502 14:14:00.811646 11624 raft_consensus.cc:348] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: true } }
I20240502 14:14:00.812134 11624 raft_consensus.cc:729] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c6030342d9b40bd91c198e3634a041a, State: Initialized, Role: LEARNER
I20240502 14:14:00.812613 11624 consensus_queue.cc:260] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: true } }
I20240502 14:14:00.813066 11643 tablet_bootstrap.cc:492] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:00.813503 11643 tablet_bootstrap.cc:492] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237: Bootstrap complete.
I20240502 14:14:00.813777 11643 ts_tablet_manager.cc:1367] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237: Time spent bootstrapping tablet: real 0.148s user 0.052s sys 0.002s
I20240502 14:14:00.813925 11623 raft_consensus.cc:348] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: true } }
I20240502 14:14:00.814206 11623 raft_consensus.cc:729] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2c6030342d9b40bd91c198e3634a041a, State: Initialized, Role: LEARNER
I20240502 14:14:00.814532 11623 consensus_queue.cc:260] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: true } }
I20240502 14:14:00.817167 11566 tablet_bootstrap.cc:492] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:00.817857 11566 tablet_bootstrap.cc:492] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e: Bootstrap complete.
I20240502 14:14:00.818238 11566 ts_tablet_manager.cc:1367] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e: Time spent bootstrapping tablet: real 0.102s user 0.040s sys 0.017s
I20240502 14:14:00.818437 11622 raft_consensus.cc:348] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: true } }
I20240502 14:14:00.818866 11622 raft_consensus.cc:729] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f7a0e54a60741298e73367e8bbf1237, State: Initialized, Role: LEARNER
I20240502 14:14:00.819382 11622 consensus_queue.cc:260] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: true } }
I20240502 14:14:00.819805 11566 raft_consensus.cc:348] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: true } }
I20240502 14:14:00.819957 11643 raft_consensus.cc:348] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: true } }
I20240502 14:14:00.820159 11643 raft_consensus.cc:729] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2f7a0e54a60741298e73367e8bbf1237, State: Initialized, Role: LEARNER
I20240502 14:14:00.820231 11566 raft_consensus.cc:729] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 33de03b632d64afe8eca2f78b08fa42e, State: Initialized, Role: LEARNER
I20240502 14:14:00.820358 11643 consensus_queue.cc:260] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: true } }
I20240502 14:14:00.820603 11566 consensus_queue.cc:260] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: true } }
I20240502 14:14:00.821599 11566 ts_tablet_manager.cc:1398] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:00.822820 9689 tablet_copy_service.cc:342] P 39deda1478724559830a8c119eae5be0: Request end of tablet copy session 33de03b632d64afe8eca2f78b08fa42e-40c7c0b7f94b42b392afd30752d26196 received from {username='slave'} at 127.6.196.196:53533
I20240502 14:14:00.822978 9689 tablet_copy_service.cc:433] P 39deda1478724559830a8c119eae5be0: ending tablet copy session 33de03b632d64afe8eca2f78b08fa42e-40c7c0b7f94b42b392afd30752d26196 on tablet 40c7c0b7f94b42b392afd30752d26196 with peer 33de03b632d64afe8eca2f78b08fa42e
I20240502 14:14:00.823911 11622 ts_tablet_manager.cc:1398] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237: Time spent starting tablet: real 0.031s user 0.004s sys 0.008s
I20240502 14:14:00.823920 11624 ts_tablet_manager.cc:1398] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a: Time spent starting tablet: real 0.020s user 0.005s sys 0.009s
I20240502 14:14:00.824682 11623 ts_tablet_manager.cc:1398] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a: Time spent starting tablet: real 0.018s user 0.001s sys 0.004s
I20240502 14:14:00.825773 9689 tablet_copy_service.cc:342] P 39deda1478724559830a8c119eae5be0: Request end of tablet copy session 2f7a0e54a60741298e73367e8bbf1237-db4824dd6aa44ad7bd2890238e006975 received from {username='slave'} at 127.6.196.197:41027
I20240502 14:14:00.825891 9689 tablet_copy_service.cc:433] P 39deda1478724559830a8c119eae5be0: ending tablet copy session 2f7a0e54a60741298e73367e8bbf1237-db4824dd6aa44ad7bd2890238e006975 on tablet db4824dd6aa44ad7bd2890238e006975 with peer 2f7a0e54a60741298e73367e8bbf1237
I20240502 14:14:00.828467 9689 tablet_copy_service.cc:342] P 39deda1478724559830a8c119eae5be0: Request end of tablet copy session 2c6030342d9b40bd91c198e3634a041a-6adbf15fbe4f49c5b10a30fd608e058a received from {username='slave'} at 127.6.196.195:60331
I20240502 14:14:00.828454 9553 tablet_copy_service.cc:342] P 5968a421103245be945b2453edc11c01: Request end of tablet copy session 2c6030342d9b40bd91c198e3634a041a-9a2af91313e0451391e05d10dce3c2ea received from {username='slave'} at 127.6.196.195:43031
I20240502 14:14:00.828634 9689 tablet_copy_service.cc:433] P 39deda1478724559830a8c119eae5be0: ending tablet copy session 2c6030342d9b40bd91c198e3634a041a-6adbf15fbe4f49c5b10a30fd608e058a on tablet 6adbf15fbe4f49c5b10a30fd608e058a with peer 2c6030342d9b40bd91c198e3634a041a
I20240502 14:14:00.828686 9553 tablet_copy_service.cc:433] P 5968a421103245be945b2453edc11c01: ending tablet copy session 2c6030342d9b40bd91c198e3634a041a-9a2af91313e0451391e05d10dce3c2ea on tablet 9a2af91313e0451391e05d10dce3c2ea with peer 2c6030342d9b40bd91c198e3634a041a
I20240502 14:14:00.829335 11643 ts_tablet_manager.cc:1398] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237: Time spent starting tablet: real 0.015s user 0.002s sys 0.001s
I20240502 14:14:00.832132 11633 tablet_bootstrap.cc:492] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:00.832631 11633 tablet_bootstrap.cc:492] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739: Bootstrap complete.
I20240502 14:14:00.832981 11633 ts_tablet_manager.cc:1367] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739: Time spent bootstrapping tablet: real 0.161s user 0.043s sys 0.006s
I20240502 14:14:00.834148 9689 tablet_copy_service.cc:342] P 39deda1478724559830a8c119eae5be0: Request end of tablet copy session 2f7a0e54a60741298e73367e8bbf1237-4ae2df4e083e4205a815a77897710ab1 received from {username='slave'} at 127.6.196.197:41027
I20240502 14:14:00.834255 9689 tablet_copy_service.cc:433] P 39deda1478724559830a8c119eae5be0: ending tablet copy session 2f7a0e54a60741298e73367e8bbf1237-4ae2df4e083e4205a815a77897710ab1 on tablet 4ae2df4e083e4205a815a77897710ab1 with peer 2f7a0e54a60741298e73367e8bbf1237
I20240502 14:14:00.834403 11633 raft_consensus.cc:348] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: true } }
I20240502 14:14:00.834913 11633 raft_consensus.cc:729] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b4b9d2cad7e04837bc20566099917739, State: Initialized, Role: LEARNER
I20240502 14:14:00.835253 11633 consensus_queue.cc:260] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: true } }
I20240502 14:14:00.836149 11633 ts_tablet_manager.cc:1398] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739: Time spent starting tablet: real 0.003s user 0.003s sys 0.003s
I20240502 14:14:00.836872 9553 tablet_copy_service.cc:342] P 5968a421103245be945b2453edc11c01: Request end of tablet copy session b4b9d2cad7e04837bc20566099917739-63746cf04e6145749291b0c9e28b4b82 received from {username='slave'} at 127.6.196.199:39887
I20240502 14:14:00.837036 9553 tablet_copy_service.cc:433] P 5968a421103245be945b2453edc11c01: ending tablet copy session b4b9d2cad7e04837bc20566099917739-63746cf04e6145749291b0c9e28b4b82 on tablet 63746cf04e6145749291b0c9e28b4b82 with peer b4b9d2cad7e04837bc20566099917739
I20240502 14:14:00.910349 11328 raft_consensus.cc:1203] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:00.925827 10918 raft_consensus.cc:1203] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:00.954713 11328 raft_consensus.cc:1203] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:00.957053 10782 raft_consensus.cc:1203] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:00.961441 11050 raft_consensus.cc:1203] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:01.010187 11050 raft_consensus.cc:1203] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:01.046031 10782 raft_consensus.cc:1203] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:01.133633 11184 raft_consensus.cc:1203] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:01.195920 10918 raft_consensus.cc:1203] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:01.344969 11730 raft_consensus.cc:1050] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01: attempting to promote NON_VOTER 33de03b632d64afe8eca2f78b08fa42e to VOTER
I20240502 14:14:01.345767 11730 consensus_queue.cc:237] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } }
I20240502 14:14:01.347482 10918 raft_consensus.cc:1261] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [term 1 LEARNER]: Refusing update from remote peer 5968a421103245be945b2453edc11c01: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:01.348042 11729 consensus_queue.cc:1035] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:01.351032 11730 raft_consensus.cc:2932] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } } }
I20240502 14:14:01.351671 10918 raft_consensus.cc:2932] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } } }
I20240502 14:14:01.354976 9382 catalog_manager.cc:5484] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 reported cstate change: config changed from index 4 to 5, 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:01.417802 9532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82e72d3de4e34731b7044de67bf6d635"
dest_uuid: "5968a421103245be945b2453edc11c01"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33254
I20240502 14:14:01.418090 9532 raft_consensus.cc:595] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:01.463155 11729 raft_consensus.cc:1050] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01: attempting to promote NON_VOTER b4b9d2cad7e04837bc20566099917739 to VOTER
I20240502 14:14:01.463618 11729 consensus_queue.cc:237] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } }
I20240502 14:14:01.470358 11328 raft_consensus.cc:1261] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [term 1 LEARNER]: Refusing update from remote peer 5968a421103245be945b2453edc11c01: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:01.470847 11728 consensus_queue.cc:1035] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:01.473637 11729 raft_consensus.cc:2932] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b4b9d2cad7e04837bc20566099917739 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } } }
I20240502 14:14:01.474412 11328 raft_consensus.cc:2932] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b4b9d2cad7e04837bc20566099917739 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } } }
I20240502 14:14:01.477739 9381 catalog_manager.cc:5484] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 reported cstate change: config changed from index 4 to 5, b4b9d2cad7e04837bc20566099917739 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:01.480604 11728 raft_consensus.cc:1050] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01: attempting to promote NON_VOTER b4b9d2cad7e04837bc20566099917739 to VOTER
I20240502 14:14:01.481266 11728 consensus_queue.cc:237] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } }
I20240502 14:14:01.483050 11328 raft_consensus.cc:1261] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [term 1 LEARNER]: Refusing update from remote peer 5968a421103245be945b2453edc11c01: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:01.483453 11729 consensus_queue.cc:1035] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:01.485922 11728 raft_consensus.cc:2932] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b4b9d2cad7e04837bc20566099917739 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } } }
I20240502 14:14:01.486493 11328 raft_consensus.cc:2932] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, b4b9d2cad7e04837bc20566099917739 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } } }
I20240502 14:14:01.489738 9382 catalog_manager.cc:5484] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 reported cstate change: config changed from index 4 to 5, b4b9d2cad7e04837bc20566099917739 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:01.500965 11732 raft_consensus.cc:1050] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0: attempting to promote NON_VOTER 2f7a0e54a60741298e73367e8bbf1237 to VOTER
I20240502 14:14:01.501960 11732 consensus_queue.cc:237] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } }
I20240502 14:14:01.503937 11050 raft_consensus.cc:1261] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 LEARNER]: Refusing update from remote peer 39deda1478724559830a8c119eae5be0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:01.504551 11731 consensus_queue.cc:1035] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:01.507824 11732 raft_consensus.cc:2932] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } } }
I20240502 14:14:01.508476 11050 raft_consensus.cc:2932] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } } }
I20240502 14:14:01.512493 9381 catalog_manager.cc:5484] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 reported cstate change: config changed from index 4 to 5, 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:01.514562 11732 raft_consensus.cc:1050] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0: attempting to promote NON_VOTER 2f7a0e54a60741298e73367e8bbf1237 to VOTER
I20240502 14:14:01.515264 11732 consensus_queue.cc:237] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } }
I20240502 14:14:01.516872 11050 raft_consensus.cc:1261] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 LEARNER]: Refusing update from remote peer 39deda1478724559830a8c119eae5be0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:01.517316 11520 consensus_queue.cc:1035] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:01.519439 11732 raft_consensus.cc:2932] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } } }
I20240502 14:14:01.520026 11050 raft_consensus.cc:2932] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } } }
I20240502 14:14:01.522991 9382 catalog_manager.cc:5484] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 reported cstate change: config changed from index 4 to 5, 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:01.549846 11728 raft_consensus.cc:1050] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01: attempting to promote NON_VOTER 99b8f6d3810b4240ba87893565a08ce9 to VOTER
I20240502 14:14:01.550377 11728 consensus_queue.cc:237] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: false } }
I20240502 14:14:01.552215 11184 raft_consensus.cc:1261] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 LEARNER]: Refusing update from remote peer 5968a421103245be945b2453edc11c01: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:01.552682 11729 consensus_queue.cc:1035] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:01.555130 11728 raft_consensus.cc:2932] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99b8f6d3810b4240ba87893565a08ce9 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: false } } }
I20240502 14:14:01.555763 11184 raft_consensus.cc:2932] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 99b8f6d3810b4240ba87893565a08ce9 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: false } } }
I20240502 14:14:01.559288 9381 catalog_manager.cc:5484] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 reported cstate change: config changed from index 4 to 5, 99b8f6d3810b4240ba87893565a08ce9 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:01.566241 11729 raft_consensus.cc:1050] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01: attempting to promote NON_VOTER 2c6030342d9b40bd91c198e3634a041a to VOTER
I20240502 14:14:01.566687 11729 consensus_queue.cc:237] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } }
I20240502 14:14:01.568488 10782 raft_consensus.cc:1261] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [term 1 LEARNER]: Refusing update from remote peer 5968a421103245be945b2453edc11c01: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:01.569000 11729 consensus_queue.cc:1035] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:01.572497 11730 raft_consensus.cc:2932] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c6030342d9b40bd91c198e3634a041a (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } } }
I20240502 14:14:01.573132 10782 raft_consensus.cc:2932] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c6030342d9b40bd91c198e3634a041a (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } } }
I20240502 14:14:01.576817 9382 catalog_manager.cc:5484] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 reported cstate change: config changed from index 4 to 5, 2c6030342d9b40bd91c198e3634a041a (127.6.196.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:01.647126 11731 raft_consensus.cc:1050] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0: attempting to promote NON_VOTER 2c6030342d9b40bd91c198e3634a041a to VOTER
I20240502 14:14:01.647894 11731 consensus_queue.cc:237] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } }
I20240502 14:14:01.650524 10782 raft_consensus.cc:1261] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [term 1 LEARNER]: Refusing update from remote peer 39deda1478724559830a8c119eae5be0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:01.651062 11520 consensus_queue.cc:1035] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:01.654088 11731 raft_consensus.cc:2932] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c6030342d9b40bd91c198e3634a041a (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } } }
I20240502 14:14:01.654700 10782 raft_consensus.cc:2932] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 2c6030342d9b40bd91c198e3634a041a (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } } }
I20240502 14:14:01.660370 9381 catalog_manager.cc:5484] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 reported cstate change: config changed from index 4 to 5, 2c6030342d9b40bd91c198e3634a041a (127.6.196.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:01.683601 9668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db4824dd6aa44ad7bd2890238e006975"
dest_uuid: "39deda1478724559830a8c119eae5be0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55344
I20240502 14:14:01.683842 9668 raft_consensus.cc:595] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:01.698441 9532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "63746cf04e6145749291b0c9e28b4b82"
dest_uuid: "5968a421103245be945b2453edc11c01"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33310
I20240502 14:14:01.698613 9532 raft_consensus.cc:595] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:01.709307 11520 raft_consensus.cc:1050] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0: attempting to promote NON_VOTER 33de03b632d64afe8eca2f78b08fa42e to VOTER
I20240502 14:14:01.709793 11520 consensus_queue.cc:237] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } }
I20240502 14:14:01.711467 10918 raft_consensus.cc:1261] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [term 1 LEARNER]: Refusing update from remote peer 39deda1478724559830a8c119eae5be0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:01.712008 11731 consensus_queue.cc:1035] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:01.713554 9532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "82e72d3de4e34731b7044de67bf6d635"
dest_uuid: "5968a421103245be945b2453edc11c01"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33326
I20240502 14:14:01.713712 9532 raft_consensus.cc:595] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:01.714319 11520 raft_consensus.cc:2932] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } } }
I20240502 14:14:01.714910 10918 raft_consensus.cc:2932] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } } }
I20240502 14:14:01.718747 9382 catalog_manager.cc:5484] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 reported cstate change: config changed from index 4 to 5, 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:01.732820 9532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a2af91313e0451391e05d10dce3c2ea"
dest_uuid: "5968a421103245be945b2453edc11c01"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33344
I20240502 14:14:01.732956 9532 raft_consensus.cc:595] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:01.747943 9668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6adbf15fbe4f49c5b10a30fd608e058a"
dest_uuid: "39deda1478724559830a8c119eae5be0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55370
I20240502 14:14:01.748124 9668 raft_consensus.cc:595] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:01.761164 9668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "40c7c0b7f94b42b392afd30752d26196"
dest_uuid: "39deda1478724559830a8c119eae5be0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55386
I20240502 14:14:01.761322 9668 raft_consensus.cc:595] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:01.775146 9532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c8aba452497f40ec971f5784d3edbfd1"
dest_uuid: "5968a421103245be945b2453edc11c01"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33358
I20240502 14:14:01.775283 9532 raft_consensus.cc:595] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:01.790602 9668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ae2df4e083e4205a815a77897710ab1"
dest_uuid: "39deda1478724559830a8c119eae5be0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55404
I20240502 14:14:01.790724 9668 raft_consensus.cc:595] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:01.804301 9532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ffe9dcb04b354f61a43df073c78aec59"
dest_uuid: "5968a421103245be945b2453edc11c01"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33372
I20240502 14:14:01.804457 9532 raft_consensus.cc:595] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:01.885535 11730 raft_consensus.cc:979] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01: : Instructing follower 33de03b632d64afe8eca2f78b08fa42e to start an election
I20240502 14:14:01.885679 11730 raft_consensus.cc:1067] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Signalling peer 33de03b632d64afe8eca2f78b08fa42e to start an election
I20240502 14:14:01.886397 10918 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "82e72d3de4e34731b7044de67bf6d635"
dest_uuid: "33de03b632d64afe8eca2f78b08fa42e"
from {username='slave'} at 127.6.196.193:56285
I20240502 14:14:01.886638 10918 raft_consensus.cc:482] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:14:01.886794 10918 raft_consensus.cc:3037] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:01.888474 10918 raft_consensus.cc:504] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } }
I20240502 14:14:01.889528 10918 leader_election.cc:288] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [CANDIDATE]: Term 2 election: Requested vote from peers 5968a421103245be945b2453edc11c01 (127.6.196.193:33389)
I20240502 14:14:01.890036 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "82e72d3de4e34731b7044de67bf6d635" candidate_uuid: "33de03b632d64afe8eca2f78b08fa42e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5968a421103245be945b2453edc11c01"
I20240502 14:14:01.890266 9532 raft_consensus.cc:3032] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:14:01.890332 9532 raft_consensus.cc:729] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:14:01.890585 9532 consensus_queue.cc:260] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } }
I20240502 14:14:01.890843 9532 raft_consensus.cc:3037] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:01.892494 9532 raft_consensus.cc:2454] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33de03b632d64afe8eca2f78b08fa42e in term 2.
I20240502 14:14:01.892938 10852 leader_election.cc:302] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 33de03b632d64afe8eca2f78b08fa42e, 5968a421103245be945b2453edc11c01; no voters:
I20240502 14:14:01.893247 11621 raft_consensus.cc:2781] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:01.893604 11621 raft_consensus.cc:686] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [term 2 LEADER]: Becoming Leader. State: Replica: 33de03b632d64afe8eca2f78b08fa42e, State: Running, Role: LEADER
I20240502 14:14:01.894023 11621 consensus_queue.cc:237] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } }
I20240502 14:14:01.896695 9381 catalog_manager.cc:5484] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e reported cstate change: term changed from 1 to 2, leader changed from 5968a421103245be945b2453edc11c01 (127.6.196.193) to 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196). New cstate: current_term: 2 leader_uuid: "33de03b632d64afe8eca2f78b08fa42e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:14:01.905465 9376 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 82e72d3de4e34731b7044de67bf6d635 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196:32821) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 14:14:01.964689 11729 raft_consensus.cc:979] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01: : Instructing follower b4b9d2cad7e04837bc20566099917739 to start an election
I20240502 14:14:01.964876 11729 raft_consensus.cc:1067] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Signalling peer b4b9d2cad7e04837bc20566099917739 to start an election
I20240502 14:14:01.965473 11328 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "63746cf04e6145749291b0c9e28b4b82"
dest_uuid: "b4b9d2cad7e04837bc20566099917739"
from {username='slave'} at 127.6.196.193:35169
I20240502 14:14:01.965685 11328 raft_consensus.cc:482] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:14:01.965806 11328 raft_consensus.cc:3037] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:01.967411 11328 raft_consensus.cc:504] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } }
I20240502 14:14:01.968446 11328 leader_election.cc:288] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [CANDIDATE]: Term 2 election: Requested vote from peers 5968a421103245be945b2453edc11c01 (127.6.196.193:33389)
I20240502 14:14:01.968854 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63746cf04e6145749291b0c9e28b4b82" candidate_uuid: "b4b9d2cad7e04837bc20566099917739" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5968a421103245be945b2453edc11c01"
I20240502 14:14:01.969030 9532 raft_consensus.cc:3032] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:14:01.969103 9532 raft_consensus.cc:729] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:14:01.969344 9532 consensus_queue.cc:260] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } }
I20240502 14:14:01.969561 9532 raft_consensus.cc:3037] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:01.971084 9532 raft_consensus.cc:2454] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4b9d2cad7e04837bc20566099917739 in term 2.
I20240502 14:14:01.971518 11260 leader_election.cc:302] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5968a421103245be945b2453edc11c01, b4b9d2cad7e04837bc20566099917739; no voters:
I20240502 14:14:01.971815 11837 raft_consensus.cc:2781] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:01.972170 11837 raft_consensus.cc:686] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [term 2 LEADER]: Becoming Leader. State: Replica: b4b9d2cad7e04837bc20566099917739, State: Running, Role: LEADER
I20240502 14:14:01.972520 11837 consensus_queue.cc:237] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } }
I20240502 14:14:01.975227 9381 catalog_manager.cc:5484] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 reported cstate change: term changed from 1 to 2, leader changed from 5968a421103245be945b2453edc11c01 (127.6.196.193) to b4b9d2cad7e04837bc20566099917739 (127.6.196.199). New cstate: current_term: 2 leader_uuid: "b4b9d2cad7e04837bc20566099917739" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:01.991798 11730 raft_consensus.cc:979] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01: : Instructing follower b4b9d2cad7e04837bc20566099917739 to start an election
I20240502 14:14:01.991916 11730 raft_consensus.cc:1067] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Signalling peer b4b9d2cad7e04837bc20566099917739 to start an election
I20240502 14:14:01.992399 11328 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c8aba452497f40ec971f5784d3edbfd1"
dest_uuid: "b4b9d2cad7e04837bc20566099917739"
from {username='slave'} at 127.6.196.193:35169
I20240502 14:14:01.992573 11328 raft_consensus.cc:482] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:14:01.992695 11328 raft_consensus.cc:3037] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:01.994454 11328 raft_consensus.cc:504] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } }
I20240502 14:14:01.994827 11328 leader_election.cc:288] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [CANDIDATE]: Term 2 election: Requested vote from peers 5968a421103245be945b2453edc11c01 (127.6.196.193:33389)
I20240502 14:14:01.995260 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8aba452497f40ec971f5784d3edbfd1" candidate_uuid: "b4b9d2cad7e04837bc20566099917739" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5968a421103245be945b2453edc11c01"
I20240502 14:14:01.995419 9532 raft_consensus.cc:3032] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:14:01.995481 9532 raft_consensus.cc:729] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:14:01.995693 9532 consensus_queue.cc:260] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } }
I20240502 14:14:01.995893 9532 raft_consensus.cc:3037] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:01.997443 9532 raft_consensus.cc:2454] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4b9d2cad7e04837bc20566099917739 in term 2.
I20240502 14:14:01.997717 11260 leader_election.cc:302] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5968a421103245be945b2453edc11c01, b4b9d2cad7e04837bc20566099917739; no voters:
I20240502 14:14:01.997967 11837 raft_consensus.cc:2781] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:01.998099 11837 raft_consensus.cc:686] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [term 2 LEADER]: Becoming Leader. State: Replica: b4b9d2cad7e04837bc20566099917739, State: Running, Role: LEADER
I20240502 14:14:01.998291 11837 consensus_queue.cc:237] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } }
I20240502 14:14:01.999728 9381 catalog_manager.cc:5484] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 reported cstate change: term changed from 1 to 2, leader changed from 5968a421103245be945b2453edc11c01 (127.6.196.193) to b4b9d2cad7e04837bc20566099917739 (127.6.196.199). New cstate: current_term: 2 leader_uuid: "b4b9d2cad7e04837bc20566099917739" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.020498 9668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db4824dd6aa44ad7bd2890238e006975"
dest_uuid: "39deda1478724559830a8c119eae5be0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55420
I20240502 14:14:02.020648 9668 raft_consensus.cc:595] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:02.052623 9532 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9a2af91313e0451391e05d10dce3c2ea"
dest_uuid: "5968a421103245be945b2453edc11c01"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:33404
I20240502 14:14:02.052835 9532 raft_consensus.cc:595] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:02.067005 9668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6adbf15fbe4f49c5b10a30fd608e058a"
dest_uuid: "39deda1478724559830a8c119eae5be0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55434
I20240502 14:14:02.067138 9668 raft_consensus.cc:595] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:02.074775 11731 raft_consensus.cc:979] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0: : Instructing follower 2f7a0e54a60741298e73367e8bbf1237 to start an election
I20240502 14:14:02.074944 11731 raft_consensus.cc:1067] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Signalling peer 2f7a0e54a60741298e73367e8bbf1237 to start an election
I20240502 14:14:02.075860 11050 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db4824dd6aa44ad7bd2890238e006975"
dest_uuid: "2f7a0e54a60741298e73367e8bbf1237"
from {username='slave'} at 127.6.196.194:44807
I20240502 14:14:02.076102 11050 raft_consensus.cc:482] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:14:02.076300 11050 raft_consensus.cc:3037] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:02.079136 11050 raft_consensus.cc:504] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } }
I20240502 14:14:02.080595 11050 leader_election.cc:288] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [CANDIDATE]: Term 2 election: Requested vote from peers 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583)
I20240502 14:14:02.081293 9668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db4824dd6aa44ad7bd2890238e006975" candidate_uuid: "2f7a0e54a60741298e73367e8bbf1237" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "39deda1478724559830a8c119eae5be0"
I20240502 14:14:02.081905 9668 raft_consensus.cc:3032] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:14:02.082073 9668 raft_consensus.cc:729] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39deda1478724559830a8c119eae5be0, State: Running, Role: LEADER
I20240502 14:14:02.082394 9668 consensus_queue.cc:260] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } }
I20240502 14:14:02.082794 9668 raft_consensus.cc:3037] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:02.082821 9667 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "40c7c0b7f94b42b392afd30752d26196"
dest_uuid: "39deda1478724559830a8c119eae5be0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55464
I20240502 14:14:02.083063 9667 raft_consensus.cc:595] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:02.085608 9668 raft_consensus.cc:2454] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f7a0e54a60741298e73367e8bbf1237 in term 2.
I20240502 14:14:02.086077 10985 leader_election.cc:302] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2f7a0e54a60741298e73367e8bbf1237, 39deda1478724559830a8c119eae5be0; no voters:
I20240502 14:14:02.086607 12003 raft_consensus.cc:2781] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:02.086947 12003 raft_consensus.cc:686] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [term 2 LEADER]: Becoming Leader. State: Replica: 2f7a0e54a60741298e73367e8bbf1237, State: Running, Role: LEADER
I20240502 14:14:02.087441 12003 consensus_queue.cc:237] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } }
I20240502 14:14:02.090317 9381 catalog_manager.cc:5484] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 reported cstate change: term changed from 1 to 2, leader changed from 39deda1478724559830a8c119eae5be0 (127.6.196.194) to 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197). New cstate: current_term: 2 leader_uuid: "2f7a0e54a60741298e73367e8bbf1237" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.118517 9668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4ae2df4e083e4205a815a77897710ab1"
dest_uuid: "39deda1478724559830a8c119eae5be0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55482
I20240502 14:14:02.118705 9668 raft_consensus.cc:595] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:02.122433 11729 raft_consensus.cc:979] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01: : Instructing follower 2c6030342d9b40bd91c198e3634a041a to start an election
I20240502 14:14:02.122606 11729 raft_consensus.cc:1067] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Signalling peer 2c6030342d9b40bd91c198e3634a041a to start an election
I20240502 14:14:02.123437 10782 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9a2af91313e0451391e05d10dce3c2ea"
dest_uuid: "2c6030342d9b40bd91c198e3634a041a"
from {username='slave'} at 127.6.196.193:33999
I20240502 14:14:02.123678 10782 raft_consensus.cc:482] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:14:02.123873 10782 raft_consensus.cc:3037] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:02.126037 10782 raft_consensus.cc:504] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } }
I20240502 14:14:02.127236 10782 leader_election.cc:288] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [CANDIDATE]: Term 2 election: Requested vote from peers 5968a421103245be945b2453edc11c01 (127.6.196.193:33389)
I20240502 14:14:02.127653 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a2af91313e0451391e05d10dce3c2ea" candidate_uuid: "2c6030342d9b40bd91c198e3634a041a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5968a421103245be945b2453edc11c01"
I20240502 14:14:02.127810 9532 raft_consensus.cc:3032] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:14:02.127862 9532 raft_consensus.cc:729] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:14:02.128072 9532 consensus_queue.cc:260] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } }
I20240502 14:14:02.128206 9532 raft_consensus.cc:3037] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:02.129999 9532 raft_consensus.cc:2454] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6030342d9b40bd91c198e3634a041a in term 2.
I20240502 14:14:02.130453 10715 leader_election.cc:302] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2c6030342d9b40bd91c198e3634a041a, 5968a421103245be945b2453edc11c01; no voters:
I20240502 14:14:02.130800 11850 raft_consensus.cc:2781] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:02.131528 11850 raft_consensus.cc:686] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [term 2 LEADER]: Becoming Leader. State: Replica: 2c6030342d9b40bd91c198e3634a041a, State: Running, Role: LEADER
I20240502 14:14:02.131973 11850 consensus_queue.cc:237] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } }
I20240502 14:14:02.135039 9381 catalog_manager.cc:5484] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a reported cstate change: term changed from 1 to 2, leader changed from 5968a421103245be945b2453edc11c01 (127.6.196.193) to 2c6030342d9b40bd91c198e3634a041a (127.6.196.195). New cstate: current_term: 2 leader_uuid: "2c6030342d9b40bd91c198e3634a041a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.136755 11731 raft_consensus.cc:979] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0: : Instructing follower 2f7a0e54a60741298e73367e8bbf1237 to start an election
I20240502 14:14:02.136907 11731 raft_consensus.cc:1067] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Signalling peer 2f7a0e54a60741298e73367e8bbf1237 to start an election
I20240502 14:14:02.137532 11050 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ae2df4e083e4205a815a77897710ab1"
dest_uuid: "2f7a0e54a60741298e73367e8bbf1237"
from {username='slave'} at 127.6.196.194:44807
I20240502 14:14:02.137751 11050 raft_consensus.cc:482] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:14:02.137876 11050 raft_consensus.cc:3037] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:02.139940 11730 raft_consensus.cc:979] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01: : Instructing follower 99b8f6d3810b4240ba87893565a08ce9 to start an election
I20240502 14:14:02.140053 11730 raft_consensus.cc:1067] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Signalling peer 99b8f6d3810b4240ba87893565a08ce9 to start an election
I20240502 14:14:02.140352 11050 raft_consensus.cc:504] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } }
I20240502 14:14:02.140861 11050 leader_election.cc:288] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [CANDIDATE]: Term 2 election: Requested vote from peers 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583)
I20240502 14:14:02.141469 9668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ae2df4e083e4205a815a77897710ab1" candidate_uuid: "2f7a0e54a60741298e73367e8bbf1237" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "39deda1478724559830a8c119eae5be0"
I20240502 14:14:02.141732 9668 raft_consensus.cc:3032] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:14:02.141856 9668 raft_consensus.cc:729] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39deda1478724559830a8c119eae5be0, State: Running, Role: LEADER
I20240502 14:14:02.142158 9668 consensus_queue.cc:260] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } }
I20240502 14:14:02.142469 9668 raft_consensus.cc:3037] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:02.142642 11184 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ffe9dcb04b354f61a43df073c78aec59"
dest_uuid: "99b8f6d3810b4240ba87893565a08ce9"
from {username='slave'} at 127.6.196.193:58353
I20240502 14:14:02.142961 11184 raft_consensus.cc:482] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:14:02.143098 11184 raft_consensus.cc:3037] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:02.146015 9668 raft_consensus.cc:2454] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f7a0e54a60741298e73367e8bbf1237 in term 2.
I20240502 14:14:02.146525 10985 leader_election.cc:302] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2f7a0e54a60741298e73367e8bbf1237, 39deda1478724559830a8c119eae5be0; no voters:
I20240502 14:14:02.146888 12003 raft_consensus.cc:2781] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:02.147089 12003 raft_consensus.cc:686] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [term 2 LEADER]: Becoming Leader. State: Replica: 2f7a0e54a60741298e73367e8bbf1237, State: Running, Role: LEADER
I20240502 14:14:02.147250 11184 raft_consensus.cc:504] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: false } }
I20240502 14:14:02.147357 12003 consensus_queue.cc:237] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } }
I20240502 14:14:02.148557 11184 leader_election.cc:288] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [CANDIDATE]: Term 2 election: Requested vote from peers 5968a421103245be945b2453edc11c01 (127.6.196.193:33389)
I20240502 14:14:02.149286 9532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffe9dcb04b354f61a43df073c78aec59" candidate_uuid: "99b8f6d3810b4240ba87893565a08ce9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5968a421103245be945b2453edc11c01"
I20240502 14:14:02.149539 9532 raft_consensus.cc:3032] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:14:02.149631 9532 raft_consensus.cc:729] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:14:02.149942 9532 consensus_queue.cc:260] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: false } }
I20240502 14:14:02.149847 9381 catalog_manager.cc:5484] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 reported cstate change: term changed from 1 to 2, leader changed from 39deda1478724559830a8c119eae5be0 (127.6.196.194) to 2f7a0e54a60741298e73367e8bbf1237 (127.6.196.197). New cstate: current_term: 2 leader_uuid: "2f7a0e54a60741298e73367e8bbf1237" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.150182 9532 raft_consensus.cc:3037] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:02.152536 9532 raft_consensus.cc:2454] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99b8f6d3810b4240ba87893565a08ce9 in term 2.
I20240502 14:14:02.153481 11115 leader_election.cc:302] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5968a421103245be945b2453edc11c01, 99b8f6d3810b4240ba87893565a08ce9; no voters:
I20240502 14:14:02.154016 12026 raft_consensus.cc:2781] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:02.157469 12026 raft_consensus.cc:686] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [term 2 LEADER]: Becoming Leader. State: Replica: 99b8f6d3810b4240ba87893565a08ce9, State: Running, Role: LEADER
I20240502 14:14:02.157876 12026 consensus_queue.cc:237] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: false } }
I20240502 14:14:02.161036 9381 catalog_manager.cc:5484] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 reported cstate change: term changed from 1 to 2, leader changed from 5968a421103245be945b2453edc11c01 (127.6.196.193) to 99b8f6d3810b4240ba87893565a08ce9 (127.6.196.198). New cstate: current_term: 2 leader_uuid: "99b8f6d3810b4240ba87893565a08ce9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.205865 11520 raft_consensus.cc:979] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0: : Instructing follower 2c6030342d9b40bd91c198e3634a041a to start an election
I20240502 14:14:02.206025 11520 raft_consensus.cc:1067] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Signalling peer 2c6030342d9b40bd91c198e3634a041a to start an election
I20240502 14:14:02.206602 10782 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6adbf15fbe4f49c5b10a30fd608e058a"
dest_uuid: "2c6030342d9b40bd91c198e3634a041a"
from {username='slave'} at 127.6.196.194:47797
I20240502 14:14:02.206787 10782 raft_consensus.cc:482] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:14:02.206882 10782 raft_consensus.cc:3037] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:02.208519 10782 raft_consensus.cc:504] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } }
I20240502 14:14:02.208989 10782 leader_election.cc:288] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [CANDIDATE]: Term 2 election: Requested vote from peers 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583)
I20240502 14:14:02.209439 9668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6adbf15fbe4f49c5b10a30fd608e058a" candidate_uuid: "2c6030342d9b40bd91c198e3634a041a" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "39deda1478724559830a8c119eae5be0"
I20240502 14:14:02.209633 9668 raft_consensus.cc:3032] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:14:02.209825 9668 raft_consensus.cc:729] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39deda1478724559830a8c119eae5be0, State: Running, Role: LEADER
I20240502 14:14:02.210076 9668 consensus_queue.cc:260] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } }
I20240502 14:14:02.210362 9668 raft_consensus.cc:3037] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:02.212116 9668 raft_consensus.cc:2454] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c6030342d9b40bd91c198e3634a041a in term 2.
I20240502 14:14:02.212522 10716 leader_election.cc:302] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 2c6030342d9b40bd91c198e3634a041a, 39deda1478724559830a8c119eae5be0; no voters:
I20240502 14:14:02.212762 11850 raft_consensus.cc:2781] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:02.212915 11850 raft_consensus.cc:686] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [term 2 LEADER]: Becoming Leader. State: Replica: 2c6030342d9b40bd91c198e3634a041a, State: Running, Role: LEADER
I20240502 14:14:02.213135 11850 consensus_queue.cc:237] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } }
I20240502 14:14:02.214917 9381 catalog_manager.cc:5484] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a reported cstate change: term changed from 1 to 2, leader changed from 39deda1478724559830a8c119eae5be0 (127.6.196.194) to 2c6030342d9b40bd91c198e3634a041a (127.6.196.195). New cstate: current_term: 2 leader_uuid: "2c6030342d9b40bd91c198e3634a041a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.340770 11732 raft_consensus.cc:979] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0: : Instructing follower 33de03b632d64afe8eca2f78b08fa42e to start an election
I20240502 14:14:02.340927 11732 raft_consensus.cc:1067] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Signalling peer 33de03b632d64afe8eca2f78b08fa42e to start an election
I20240502 14:14:02.341552 10918 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "40c7c0b7f94b42b392afd30752d26196"
dest_uuid: "33de03b632d64afe8eca2f78b08fa42e"
from {username='slave'} at 127.6.196.194:48963
I20240502 14:14:02.341769 10918 raft_consensus.cc:482] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:14:02.341867 10918 raft_consensus.cc:3037] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:02.343735 10918 raft_consensus.cc:504] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } }
I20240502 14:14:02.344272 10918 leader_election.cc:288] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [CANDIDATE]: Term 2 election: Requested vote from peers 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583)
I20240502 14:14:02.345098 9668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40c7c0b7f94b42b392afd30752d26196" candidate_uuid: "33de03b632d64afe8eca2f78b08fa42e" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "39deda1478724559830a8c119eae5be0"
I20240502 14:14:02.345312 9668 raft_consensus.cc:3032] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:14:02.345423 9668 raft_consensus.cc:729] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39deda1478724559830a8c119eae5be0, State: Running, Role: LEADER
I20240502 14:14:02.345641 9668 consensus_queue.cc:260] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } }
I20240502 14:14:02.345928 9668 raft_consensus.cc:3037] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:02.347678 9668 raft_consensus.cc:2454] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33de03b632d64afe8eca2f78b08fa42e in term 2.
I20240502 14:14:02.348054 10853 leader_election.cc:302] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 33de03b632d64afe8eca2f78b08fa42e, 39deda1478724559830a8c119eae5be0; no voters:
I20240502 14:14:02.348287 11621 raft_consensus.cc:2781] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:02.348450 11621 raft_consensus.cc:686] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [term 2 LEADER]: Becoming Leader. State: Replica: 33de03b632d64afe8eca2f78b08fa42e, State: Running, Role: LEADER
I20240502 14:14:02.348702 11621 consensus_queue.cc:237] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } }
I20240502 14:14:02.350584 9381 catalog_manager.cc:5484] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e reported cstate change: term changed from 1 to 2, leader changed from 39deda1478724559830a8c119eae5be0 (127.6.196.194) to 33de03b632d64afe8eca2f78b08fa42e (127.6.196.196). New cstate: current_term: 2 leader_uuid: "33de03b632d64afe8eca2f78b08fa42e" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.416013 9532 raft_consensus.cc:1261] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Refusing update from remote peer b4b9d2cad7e04837bc20566099917739: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:14:02.416517 9531 raft_consensus.cc:1261] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Refusing update from remote peer 33de03b632d64afe8eca2f78b08fa42e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:14:02.416774 11837 consensus_queue.cc:1035] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:02.417088 11621 consensus_queue.cc:1035] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:02.420571 9532 raft_consensus.cc:1261] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Refusing update from remote peer b4b9d2cad7e04837bc20566099917739: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:14:02.421397 11837 consensus_queue.cc:1035] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:02.533001 10918 consensus_queue.cc:237] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } }
I20240502 14:14:02.533880 11621 raft_consensus.cc:2932] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5968a421103245be945b2453edc11c01 (127.6.196.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } } }
I20240502 14:14:02.536592 9376 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 82e72d3de4e34731b7044de67bf6d635 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:14:02.537943 9381 catalog_manager.cc:5484] T 82e72d3de4e34731b7044de67bf6d635 P 33de03b632d64afe8eca2f78b08fa42e reported cstate change: config changed from index 5 to 7, VOTER 5968a421103245be945b2453edc11c01 (127.6.196.193) evicted. New cstate: current_term: 2 leader_uuid: "33de03b632d64afe8eca2f78b08fa42e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.541441 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet 82e72d3de4e34731b7044de67bf6d635 with delete_type TABLET_DATA_TOMBSTONED (TS 5968a421103245be945b2453edc11c01 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56970
I20240502 14:14:02.542248 12084 tablet_replica.cc:330] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01: stopping tablet replica
I20240502 14:14:02.542639 12084 raft_consensus.cc:2229] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:02.542871 12084 raft_consensus.cc:2258] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:02.544360 12084 ts_tablet_manager.cc:1875] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:02.548534 12084 ts_tablet_manager.cc:1888] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:14:02.548652 12084 log.cc:1196] T 82e72d3de4e34731b7044de67bf6d635 P 5968a421103245be945b2453edc11c01: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/82e72d3de4e34731b7044de67bf6d635
I20240502 14:14:02.549427 9375 catalog_manager.cc:4830] TS 5968a421103245be945b2453edc11c01 (127.6.196.193:33389): tablet 82e72d3de4e34731b7044de67bf6d635 (table rebalance_test_table_2 [id=1af730f9592947898d4b5e6f2b642194]) successfully deleted
I20240502 14:14:02.602156 9668 raft_consensus.cc:1261] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Refusing update from remote peer 2f7a0e54a60741298e73367e8bbf1237: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:14:02.602861 12005 consensus_queue.cc:1035] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:02.617516 9668 raft_consensus.cc:1261] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Refusing update from remote peer 2c6030342d9b40bd91c198e3634a041a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:14:02.618132 11850 consensus_queue.cc:1035] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [LEADER]: Connected to new peer: Peer: permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:02.632112 11328 consensus_queue.cc:237] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } }
I20240502 14:14:02.632812 11949 raft_consensus.cc:2932] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5968a421103245be945b2453edc11c01 (127.6.196.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } } }
I20240502 14:14:02.635480 9377 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63746cf04e6145749291b0c9e28b4b82 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:14:02.636590 9381 catalog_manager.cc:5484] T 63746cf04e6145749291b0c9e28b4b82 P b4b9d2cad7e04837bc20566099917739 reported cstate change: config changed from index 5 to 7, VOTER 5968a421103245be945b2453edc11c01 (127.6.196.193) evicted. New cstate: current_term: 2 leader_uuid: "b4b9d2cad7e04837bc20566099917739" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.639390 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet 63746cf04e6145749291b0c9e28b4b82 with delete_type TABLET_DATA_TOMBSTONED (TS 5968a421103245be945b2453edc11c01 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56970
I20240502 14:14:02.639699 12084 tablet_replica.cc:330] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01: stopping tablet replica
I20240502 14:14:02.639899 12084 raft_consensus.cc:2229] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:02.640113 12084 raft_consensus.cc:2258] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:02.640782 12084 ts_tablet_manager.cc:1875] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:02.645203 12084 ts_tablet_manager.cc:1888] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:14:02.645382 12084 log.cc:1196] T 63746cf04e6145749291b0c9e28b4b82 P 5968a421103245be945b2453edc11c01: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/63746cf04e6145749291b0c9e28b4b82
I20240502 14:14:02.646005 9375 catalog_manager.cc:4830] TS 5968a421103245be945b2453edc11c01 (127.6.196.193:33389): tablet 63746cf04e6145749291b0c9e28b4b82 (table rebalance_test_table_0 [id=e2fdab079d66498eb09863f56c17fbf2]) successfully deleted
I20240502 14:14:02.651887 11328 consensus_queue.cc:237] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } }
I20240502 14:14:02.652573 11949 raft_consensus.cc:2932] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5968a421103245be945b2453edc11c01 (127.6.196.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } } }
I20240502 14:14:02.655859 9377 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c8aba452497f40ec971f5784d3edbfd1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:14:02.656956 9381 catalog_manager.cc:5484] T c8aba452497f40ec971f5784d3edbfd1 P b4b9d2cad7e04837bc20566099917739 reported cstate change: config changed from index 5 to 7, VOTER 5968a421103245be945b2453edc11c01 (127.6.196.193) evicted. New cstate: current_term: 2 leader_uuid: "b4b9d2cad7e04837bc20566099917739" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b4b9d2cad7e04837bc20566099917739" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 43261 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.660212 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet c8aba452497f40ec971f5784d3edbfd1 with delete_type TABLET_DATA_TOMBSTONED (TS 5968a421103245be945b2453edc11c01 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56970
I20240502 14:14:02.660493 12084 tablet_replica.cc:330] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01: stopping tablet replica
I20240502 14:14:02.660655 12084 raft_consensus.cc:2229] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:02.660821 12084 raft_consensus.cc:2258] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:02.661965 12084 ts_tablet_manager.cc:1875] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:02.661984 9531 raft_consensus.cc:1261] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Refusing update from remote peer 2c6030342d9b40bd91c198e3634a041a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:14:02.662525 11850 consensus_queue.cc:1035] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:02.670168 12084 ts_tablet_manager.cc:1888] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:14:02.670326 12084 log.cc:1196] T c8aba452497f40ec971f5784d3edbfd1 P 5968a421103245be945b2453edc11c01: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/c8aba452497f40ec971f5784d3edbfd1
I20240502 14:14:02.671015 9375 catalog_manager.cc:4830] TS 5968a421103245be945b2453edc11c01 (127.6.196.193:33389): tablet c8aba452497f40ec971f5784d3edbfd1 (table rebalance_test_table_4 [id=31c72864384d44d9afa69f8dcd7ef427]) successfully deleted
I20240502 14:14:02.709242 11050 consensus_queue.cc:237] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } }
I20240502 14:14:02.710112 12003 raft_consensus.cc:2932] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 39deda1478724559830a8c119eae5be0 (127.6.196.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } } }
I20240502 14:14:02.713585 9377 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db4824dd6aa44ad7bd2890238e006975 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:14:02.714715 9381 catalog_manager.cc:5484] T db4824dd6aa44ad7bd2890238e006975 P 2f7a0e54a60741298e73367e8bbf1237 reported cstate change: config changed from index 5 to 7, VOTER 39deda1478724559830a8c119eae5be0 (127.6.196.194) evicted. New cstate: current_term: 2 leader_uuid: "2f7a0e54a60741298e73367e8bbf1237" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.718600 9647 tablet_service.cc:1514] Processing DeleteTablet for tablet db4824dd6aa44ad7bd2890238e006975 with delete_type TABLET_DATA_TOMBSTONED (TS 39deda1478724559830a8c119eae5be0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45240
I20240502 14:14:02.720557 12121 tablet_replica.cc:330] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0: stopping tablet replica
I20240502 14:14:02.721045 12121 raft_consensus.cc:2229] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:02.721325 12121 raft_consensus.cc:2258] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:02.722796 12121 ts_tablet_manager.cc:1875] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:02.728775 12121 ts_tablet_manager.cc:1888] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:14:02.728937 12121 log.cc:1196] T db4824dd6aa44ad7bd2890238e006975 P 39deda1478724559830a8c119eae5be0: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/db4824dd6aa44ad7bd2890238e006975
I20240502 14:14:02.729723 9376 catalog_manager.cc:4830] TS 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583): tablet db4824dd6aa44ad7bd2890238e006975 (table rebalance_test_table_4 [id=31c72864384d44d9afa69f8dcd7ef427]) successfully deleted
I20240502 14:14:02.750394 9531 raft_consensus.cc:1261] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Refusing update from remote peer 99b8f6d3810b4240ba87893565a08ce9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:14:02.751322 12143 consensus_queue.cc:1035] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:02.754103 9668 raft_consensus.cc:1261] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Refusing update from remote peer 33de03b632d64afe8eca2f78b08fa42e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:14:02.754720 11621 consensus_queue.cc:1035] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [LEADER]: Connected to new peer: Peer: permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:02.760797 9668 raft_consensus.cc:1261] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Refusing update from remote peer 2f7a0e54a60741298e73367e8bbf1237: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:14:02.761476 12003 consensus_queue.cc:1035] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:02.779860 10782 consensus_queue.cc:237] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } }
I20240502 14:14:02.780722 12022 raft_consensus.cc:2932] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5968a421103245be945b2453edc11c01 (127.6.196.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } } }
I20240502 14:14:02.783946 9374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9a2af91313e0451391e05d10dce3c2ea with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:14:02.785190 9382 catalog_manager.cc:5484] T 9a2af91313e0451391e05d10dce3c2ea P 2c6030342d9b40bd91c198e3634a041a reported cstate change: config changed from index 5 to 7, VOTER 5968a421103245be945b2453edc11c01 (127.6.196.193) evicted. New cstate: current_term: 2 leader_uuid: "2c6030342d9b40bd91c198e3634a041a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.789341 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a2af91313e0451391e05d10dce3c2ea with delete_type TABLET_DATA_TOMBSTONED (TS 5968a421103245be945b2453edc11c01 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56970
I20240502 14:14:02.789664 12084 tablet_replica.cc:330] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01: stopping tablet replica
I20240502 14:14:02.789850 12084 raft_consensus.cc:2229] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:02.790035 12084 raft_consensus.cc:2258] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:02.790863 12084 ts_tablet_manager.cc:1875] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:02.795430 12084 ts_tablet_manager.cc:1888] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:14:02.795542 12084 log.cc:1196] T 9a2af91313e0451391e05d10dce3c2ea P 5968a421103245be945b2453edc11c01: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/9a2af91313e0451391e05d10dce3c2ea
I20240502 14:14:02.797581 11184 consensus_queue.cc:237] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: false } }
I20240502 14:14:02.798451 12143 raft_consensus.cc:2932] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5968a421103245be945b2453edc11c01 (127.6.196.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: false } } }
I20240502 14:14:02.801596 9375 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ffe9dcb04b354f61a43df073c78aec59 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:14:02.801873 9375 catalog_manager.cc:4830] TS 5968a421103245be945b2453edc11c01 (127.6.196.193:33389): tablet 9a2af91313e0451391e05d10dce3c2ea (table rebalance_test_table_1 [id=da24f830d1484a7391d1079e2ef8ca05]) successfully deleted
I20240502 14:14:02.802668 9382 catalog_manager.cc:5484] T ffe9dcb04b354f61a43df073c78aec59 P 99b8f6d3810b4240ba87893565a08ce9 reported cstate change: config changed from index 5 to 7, VOTER 5968a421103245be945b2453edc11c01 (127.6.196.193) evicted. New cstate: current_term: 2 leader_uuid: "99b8f6d3810b4240ba87893565a08ce9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "99b8f6d3810b4240ba87893565a08ce9" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 34211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.806046 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet ffe9dcb04b354f61a43df073c78aec59 with delete_type TABLET_DATA_TOMBSTONED (TS 5968a421103245be945b2453edc11c01 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56970
I20240502 14:14:02.806326 12084 tablet_replica.cc:330] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01: stopping tablet replica
I20240502 14:14:02.806476 12084 raft_consensus.cc:2229] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:02.806640 12084 raft_consensus.cc:2258] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:02.807423 12084 ts_tablet_manager.cc:1875] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:02.812527 12084 ts_tablet_manager.cc:1888] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:14:02.812655 12084 log.cc:1196] T ffe9dcb04b354f61a43df073c78aec59 P 5968a421103245be945b2453edc11c01: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/ffe9dcb04b354f61a43df073c78aec59
I20240502 14:14:02.813310 9375 catalog_manager.cc:4830] TS 5968a421103245be945b2453edc11c01 (127.6.196.193:33389): tablet ffe9dcb04b354f61a43df073c78aec59 (table rebalance_test_table_2 [id=1af730f9592947898d4b5e6f2b642194]) successfully deleted
I20240502 14:14:02.831491 10783 consensus_queue.cc:237] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } }
I20240502 14:14:02.832300 12022 raft_consensus.cc:2932] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 39deda1478724559830a8c119eae5be0 (127.6.196.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } } }
I20240502 14:14:02.834890 9374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6adbf15fbe4f49c5b10a30fd608e058a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:14:02.836011 9382 catalog_manager.cc:5484] T 6adbf15fbe4f49c5b10a30fd608e058a P 2c6030342d9b40bd91c198e3634a041a reported cstate change: config changed from index 5 to 7, VOTER 39deda1478724559830a8c119eae5be0 (127.6.196.194) evicted. New cstate: current_term: 2 leader_uuid: "2c6030342d9b40bd91c198e3634a041a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2c6030342d9b40bd91c198e3634a041a" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 41811 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:02.839680 9647 tablet_service.cc:1514] Processing DeleteTablet for tablet 6adbf15fbe4f49c5b10a30fd608e058a with delete_type TABLET_DATA_TOMBSTONED (TS 39deda1478724559830a8c119eae5be0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45240
I20240502 14:14:02.840098 12121 tablet_replica.cc:330] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0: stopping tablet replica
I20240502 14:14:02.840309 12121 raft_consensus.cc:2229] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:02.840548 12121 raft_consensus.cc:2258] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:02.841533 12121 ts_tablet_manager.cc:1875] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:02.847193 12121 ts_tablet_manager.cc:1888] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:14:02.847309 12121 log.cc:1196] T 6adbf15fbe4f49c5b10a30fd608e058a P 39deda1478724559830a8c119eae5be0: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/6adbf15fbe4f49c5b10a30fd608e058a
I20240502 14:14:02.848114 9376 catalog_manager.cc:4830] TS 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583): tablet 6adbf15fbe4f49c5b10a30fd608e058a (table rebalance_test_table_3 [id=21a0de7bef9c4ac685e4c4a9286573eb]) successfully deleted
I20240502 14:14:03.022923 10918 consensus_queue.cc:237] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } }
I20240502 14:14:03.023764 11946 raft_consensus.cc:2932] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 39deda1478724559830a8c119eae5be0 (127.6.196.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } } }
I20240502 14:14:03.026420 9376 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 40c7c0b7f94b42b392afd30752d26196 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:14:03.027776 9382 catalog_manager.cc:5484] T 40c7c0b7f94b42b392afd30752d26196 P 33de03b632d64afe8eca2f78b08fa42e reported cstate change: config changed from index 5 to 7, VOTER 39deda1478724559830a8c119eae5be0 (127.6.196.194) evicted. New cstate: current_term: 2 leader_uuid: "33de03b632d64afe8eca2f78b08fa42e" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "33de03b632d64afe8eca2f78b08fa42e" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 32821 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:03.031188 9647 tablet_service.cc:1514] Processing DeleteTablet for tablet 40c7c0b7f94b42b392afd30752d26196 with delete_type TABLET_DATA_TOMBSTONED (TS 39deda1478724559830a8c119eae5be0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45240
I20240502 14:14:03.031526 12121 tablet_replica.cc:330] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0: stopping tablet replica
I20240502 14:14:03.031699 12121 raft_consensus.cc:2229] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:03.031891 12121 raft_consensus.cc:2258] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:03.032701 12121 ts_tablet_manager.cc:1875] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:03.036861 12121 ts_tablet_manager.cc:1888] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:14:03.036953 12121 log.cc:1196] T 40c7c0b7f94b42b392afd30752d26196 P 39deda1478724559830a8c119eae5be0: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/40c7c0b7f94b42b392afd30752d26196
I20240502 14:14:03.037585 9376 catalog_manager.cc:4830] TS 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583): tablet 40c7c0b7f94b42b392afd30752d26196 (table rebalance_test_table_3 [id=21a0de7bef9c4ac685e4c4a9286573eb]) successfully deleted
I20240502 14:14:03.328490 11050 consensus_queue.cc:237] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } }
I20240502 14:14:03.329279 12222 raft_consensus.cc:2932] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 39deda1478724559830a8c119eae5be0 (127.6.196.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } } }
I20240502 14:14:03.331784 9377 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4ae2df4e083e4205a815a77897710ab1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 7)
I20240502 14:14:03.332963 9382 catalog_manager.cc:5484] T 4ae2df4e083e4205a815a77897710ab1 P 2f7a0e54a60741298e73367e8bbf1237 reported cstate change: config changed from index 5 to 7, VOTER 39deda1478724559830a8c119eae5be0 (127.6.196.194) evicted. New cstate: current_term: 2 leader_uuid: "2f7a0e54a60741298e73367e8bbf1237" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "2f7a0e54a60741298e73367e8bbf1237" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 45913 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:03.336153 9647 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ae2df4e083e4205a815a77897710ab1 with delete_type TABLET_DATA_TOMBSTONED (TS 39deda1478724559830a8c119eae5be0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45240
I20240502 14:14:03.336469 12121 tablet_replica.cc:330] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0: stopping tablet replica
I20240502 14:14:03.336656 12121 raft_consensus.cc:2229] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:03.336896 12121 raft_consensus.cc:2258] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:03.337700 12121 ts_tablet_manager.cc:1875] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:03.342145 12121 ts_tablet_manager.cc:1888] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:14:03.342259 12121 log.cc:1196] T 4ae2df4e083e4205a815a77897710ab1 P 39deda1478724559830a8c119eae5be0: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/4ae2df4e083e4205a815a77897710ab1
I20240502 14:14:03.342897 9376 catalog_manager.cc:4830] TS 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583): tablet 4ae2df4e083e4205a815a77897710ab1 (table rebalance_test_table_1 [id=da24f830d1484a7391d1079e2ef8ca05]) successfully deleted
I20240502 14:14:03.624917 11030 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:03.636251 10898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:03.668773 9647 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:03.669499 10762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:03.670500 9511 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:03.684484 11163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:03.720283 11308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:03.907538 10762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:03.911255 9511 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:03.918675 10898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:03.935633 11308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:03.936524 9647 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:03.953397 11030 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:03.968873 11163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.652144 11030 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.652177 10762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.662207 9511 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:04.699981 11308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.710520 10898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.710981 11163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.711488 9647 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:04.870401 10762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.872162 11030 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.884091 10898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.905400 9647 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:04.909947 9511 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:04.920295 11163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:04.923261 11308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:05.177960 10762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:05.196534 9511 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:05.201792 9647 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:05.203747 10898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:05.216784 11030 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:05.231195 11163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:05.255254 11308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
247adb9a8de14791b6b2872e02aeff89 | 127.6.196.254:43585 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.6.196.212:43875 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2c6030342d9b40bd91c198e3634a041a | 127.6.196.195:41811 | HEALTHY | <none> | 2 | 0
2f7a0e54a60741298e73367e8bbf1237 | 127.6.196.197:45913 | HEALTHY | <none> | 2 | 0
33de03b632d64afe8eca2f78b08fa42e | 127.6.196.196:32821 | HEALTHY | <none> | 2 | 0
39deda1478724559830a8c119eae5be0 | 127.6.196.194:35583 | HEALTHY | <none> | 3 | 0
5968a421103245be945b2453edc11c01 | 127.6.196.193:33389 | HEALTHY | <none> | 3 | 0
99b8f6d3810b4240ba87893565a08ce9 | 127.6.196.198:34211 | HEALTHY | <none> | 2 | 0
b4b9d2cad7e04837bc20566099917739 | 127.6.196.199:43261 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.6.196.193 | experimental | 127.6.196.193:33389
local_ip_for_outbound_sockets | 127.6.196.194 | experimental | 127.6.196.194:35583
local_ip_for_outbound_sockets | 127.6.196.195 | experimental | 127.6.196.195:41811
local_ip_for_outbound_sockets | 127.6.196.196 | experimental | 127.6.196.196:32821
local_ip_for_outbound_sockets | 127.6.196.197 | experimental | 127.6.196.197:45913
local_ip_for_outbound_sockets | 127.6.196.198 | experimental | 127.6.196.198:34211
local_ip_for_outbound_sockets | 127.6.196.199 | experimental | 127.6.196.199:43261
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.6.196.193:33389
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.6.196.194:35583
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.6.196.195:41811
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.6.196.196:32821
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.6.196.197:45913
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.6.196.198:34211
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.6.196.199:43261
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.6.196.212:43875 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:14:05.304028 6931 log_verifier.cc:126] Checking tablet 40c7c0b7f94b42b392afd30752d26196
I20240502 14:14:05.319865 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 40c7c0b7f94b42b392afd30752d26196
I20240502 14:14:05.320035 6931 log_verifier.cc:126] Checking tablet 4ae2df4e083e4205a815a77897710ab1
I20240502 14:14:05.332520 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 4ae2df4e083e4205a815a77897710ab1
I20240502 14:14:05.332634 6931 log_verifier.cc:126] Checking tablet 594efe32e19c46658945080d250ca29c
I20240502 14:14:05.345244 6931 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 594efe32e19c46658945080d250ca29c
I20240502 14:14:05.345363 6931 log_verifier.cc:126] Checking tablet 5f8493b9da2c4cf29de9a1ac4c70cac0
I20240502 14:14:05.352947 6931 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 5f8493b9da2c4cf29de9a1ac4c70cac0
I20240502 14:14:05.353052 6931 log_verifier.cc:126] Checking tablet 63746cf04e6145749291b0c9e28b4b82
I20240502 14:14:05.365132 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 63746cf04e6145749291b0c9e28b4b82
I20240502 14:14:05.365257 6931 log_verifier.cc:126] Checking tablet 6adbf15fbe4f49c5b10a30fd608e058a
I20240502 14:14:05.377425 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6adbf15fbe4f49c5b10a30fd608e058a
I20240502 14:14:05.377534 6931 log_verifier.cc:126] Checking tablet 7d4a89cb247c4b8ea952c245ca6f6377
I20240502 14:14:05.389099 6931 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7d4a89cb247c4b8ea952c245ca6f6377
I20240502 14:14:05.389199 6931 log_verifier.cc:126] Checking tablet 828f6ad5260a469384cad06dd2986b05
I20240502 14:14:05.400713 6931 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 828f6ad5260a469384cad06dd2986b05
I20240502 14:14:05.400810 6931 log_verifier.cc:126] Checking tablet 82e72d3de4e34731b7044de67bf6d635
I20240502 14:14:05.415741 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 82e72d3de4e34731b7044de67bf6d635
I20240502 14:14:05.415854 6931 log_verifier.cc:126] Checking tablet 93d3db61bd8143d48d66ce379e443323
I20240502 14:14:05.428812 6931 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 93d3db61bd8143d48d66ce379e443323
I20240502 14:14:05.428917 6931 log_verifier.cc:126] Checking tablet 9a2af91313e0451391e05d10dce3c2ea
I20240502 14:14:05.445732 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9a2af91313e0451391e05d10dce3c2ea
I20240502 14:14:05.445839 6931 log_verifier.cc:126] Checking tablet c8aba452497f40ec971f5784d3edbfd1
I20240502 14:14:05.459619 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c8aba452497f40ec971f5784d3edbfd1
I20240502 14:14:05.459723 6931 log_verifier.cc:126] Checking tablet cb6cacc3e4564116b13c696813ce0426
I20240502 14:14:05.472258 6931 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cb6cacc3e4564116b13c696813ce0426
I20240502 14:14:05.472362 6931 log_verifier.cc:126] Checking tablet cf42265c4952414d90b9ac9d8fb934e6
I20240502 14:14:05.485008 6931 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cf42265c4952414d90b9ac9d8fb934e6
I20240502 14:14:05.485103 6931 log_verifier.cc:126] Checking tablet db4824dd6aa44ad7bd2890238e006975
I20240502 14:14:05.500916 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet db4824dd6aa44ad7bd2890238e006975
I20240502 14:14:05.501031 6931 log_verifier.cc:126] Checking tablet ffe9dcb04b354f61a43df073c78aec59
I20240502 14:14:05.513818 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ffe9dcb04b354f61a43df073c78aec59
I20240502 14:14:05.516681 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.200:0
--local_ip_for_outbound_sockets=127.6.196.200
--webserver_interface=127.6.196.200
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--builtin_ntp_servers=127.6.196.212:43875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:14:05.839684 12421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:05.839972 12421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:05.840112 12421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:05.848402 12421 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:05.848523 12421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:05.848699 12421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.200
I20240502 14:14:05.857795 12421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:43875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.196.200
--webserver_port=0
--tserver_master_addrs=127.6.196.254:43585
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.200
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:05.859086 12421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:05.860939 12421 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:05.873208 12426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:05.873746 12427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:05.874329 12429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:05.875293 12421 server_base.cc:1034] running on GCE node
I20240502 14:14:05.875857 12421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:05.876683 12421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:05.877897 12421 hybrid_clock.cc:648] HybridClock initialized: now 1714659245877839 us; error 40 us; skew 500 ppm
I20240502 14:14:05.878230 12421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:05.880440 12421 webserver.cc:460] Webserver started at http://127.6.196.200:38783/ using document root <none> and password file <none>
I20240502 14:14:05.881182 12421 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:05.881306 12421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:05.881613 12421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:05.883950 12421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "d9e7b81687664b5f8009584f65b45847"
format_stamp: "Formatted at 2024-05-02 14:14:05 on dist-test-slave-kjf7"
I20240502 14:14:05.884699 12421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "d9e7b81687664b5f8009584f65b45847"
format_stamp: "Formatted at 2024-05-02 14:14:05 on dist-test-slave-kjf7"
I20240502 14:14:05.889379 12421 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20240502 14:14:05.892356 12435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:05.893543 12421 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:05.893687 12421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "d9e7b81687664b5f8009584f65b45847"
format_stamp: "Formatted at 2024-05-02 14:14:05 on dist-test-slave-kjf7"
I20240502 14:14:05.893859 12421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:05.913398 12421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:05.914078 12421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:05.914376 12421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:05.915216 12421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:05.916561 12421 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:05.916656 12421 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:05.916766 12421 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:05.916847 12421 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:05.948068 12421 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.200:32911
I20240502 14:14:05.948194 12549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.200:32911 every 8 connection(s)
I20240502 14:14:05.949191 12421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:14:05.955610 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 12421
I20240502 14:14:05.955833 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:14:05.961273 12550 heartbeater.cc:347] Connected to a master server at 127.6.196.254:43585
I20240502 14:14:05.961524 12550 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:05.962226 12550 heartbeater.cc:511] Master 127.6.196.254:43585 requested a full tablet report, sending...
I20240502 14:14:05.963508 9382 ts_manager.cc:194] Registered new tserver with Master: d9e7b81687664b5f8009584f65b45847 (127.6.196.200:32911)
I20240502 14:14:05.964207 9382 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.200:54551
I20240502 14:14:06.395258 10762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:06.396189 11030 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:06.426308 10898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:06.438627 9647 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:06.438866 9511 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:06.453255 11163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:06.471429 11308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:06.476154 12485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:06.646941 10762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:06.649704 11030 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:06.673029 10898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:06.684530 9647 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:06.708922 9511 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:06.720155 11308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:06.737269 12485 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:06.742623 11163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:06.800824 9531 consensus_queue.cc:237] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: true } }
I20240502 14:14:06.802695 12644 raft_consensus.cc:2932] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d9e7b81687664b5f8009584f65b45847 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: true } } }
I20240502 14:14:06.809335 9386 catalog_manager.cc:5484] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 reported cstate change: config changed from index -1 to 4, NON_VOTER d9e7b81687664b5f8009584f65b45847 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:06.825057 9465 consensus_peers.cc:485] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 -> Peer d9e7b81687664b5f8009584f65b45847 (127.6.196.200:32911): Couldn't send request to peer d9e7b81687664b5f8009584f65b45847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb6cacc3e4564116b13c696813ce0426. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:06.834950 9668 consensus_queue.cc:237] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: true } }
I20240502 14:14:06.836709 12657 raft_consensus.cc:2932] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d9e7b81687664b5f8009584f65b45847 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: true } } }
I20240502 14:14:06.842063 9386 catalog_manager.cc:5484] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 reported cstate change: config changed from index -1 to 4, NON_VOTER d9e7b81687664b5f8009584f65b45847 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:06.845455 9598 consensus_peers.cc:485] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 -> Peer d9e7b81687664b5f8009584f65b45847 (127.6.196.200:32911): Couldn't send request to peer d9e7b81687664b5f8009584f65b45847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 828f6ad5260a469384cad06dd2986b05. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:06.966284 12550 heartbeater.cc:503] Master 127.6.196.254:43585 was elected leader, sending a full tablet report...
I20240502 14:14:07.243546 12680 ts_tablet_manager.cc:902] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847: Initiating tablet copy from peer 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583)
I20240502 14:14:07.245570 12680 tablet_copy_client.cc:308] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:35583
I20240502 14:14:07.251396 9689 tablet_copy_service.cc:140] P 39deda1478724559830a8c119eae5be0: Received BeginTabletCopySession request for tablet 828f6ad5260a469384cad06dd2986b05 from peer d9e7b81687664b5f8009584f65b45847 ({username='slave'} at 127.6.196.200:45723)
I20240502 14:14:07.251520 9689 tablet_copy_service.cc:161] P 39deda1478724559830a8c119eae5be0: Beginning new tablet copy session on tablet 828f6ad5260a469384cad06dd2986b05 from peer d9e7b81687664b5f8009584f65b45847 at {username='slave'} at 127.6.196.200:45723: session id = d9e7b81687664b5f8009584f65b45847-828f6ad5260a469384cad06dd2986b05
I20240502 14:14:07.253307 9689 tablet_copy_source_session.cc:205] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:07.257212 12680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 828f6ad5260a469384cad06dd2986b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:07.266227 12680 tablet_copy_client.cc:783] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:07.266537 12680 tablet_copy_client.cc:647] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:07.268435 12680 tablet_copy_client.cc:523] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:07.271209 12680 tablet_bootstrap.cc:492] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847: Bootstrap starting.
I20240502 14:14:07.278798 12680 log.cc:824] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:07.310037 12680 tablet_bootstrap.cc:492] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:07.310470 12680 tablet_bootstrap.cc:492] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847: Bootstrap complete.
I20240502 14:14:07.310853 12680 ts_tablet_manager.cc:1367] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847: Time spent bootstrapping tablet: real 0.040s user 0.031s sys 0.008s
I20240502 14:14:07.314950 12697 ts_tablet_manager.cc:902] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847: Initiating tablet copy from peer 5968a421103245be945b2453edc11c01 (127.6.196.193:33389)
I20240502 14:14:07.315755 12680 raft_consensus.cc:348] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: true } }
I20240502 14:14:07.316318 12680 raft_consensus.cc:729] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d9e7b81687664b5f8009584f65b45847, State: Initialized, Role: LEARNER
I20240502 14:14:07.316848 12680 consensus_queue.cc:260] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: true } }
I20240502 14:14:07.317472 12697 tablet_copy_client.cc:308] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:33389
I20240502 14:14:07.318728 12680 ts_tablet_manager.cc:1398] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847: Time spent starting tablet: real 0.008s user 0.006s sys 0.002s
I20240502 14:14:07.319622 9689 tablet_copy_service.cc:342] P 39deda1478724559830a8c119eae5be0: Request end of tablet copy session d9e7b81687664b5f8009584f65b45847-828f6ad5260a469384cad06dd2986b05 received from {username='slave'} at 127.6.196.200:45723
I20240502 14:14:07.319777 9689 tablet_copy_service.cc:433] P 39deda1478724559830a8c119eae5be0: ending tablet copy session d9e7b81687664b5f8009584f65b45847-828f6ad5260a469384cad06dd2986b05 on tablet 828f6ad5260a469384cad06dd2986b05 with peer d9e7b81687664b5f8009584f65b45847
I20240502 14:14:07.324337 9553 tablet_copy_service.cc:140] P 5968a421103245be945b2453edc11c01: Received BeginTabletCopySession request for tablet cb6cacc3e4564116b13c696813ce0426 from peer d9e7b81687664b5f8009584f65b45847 ({username='slave'} at 127.6.196.200:55757)
I20240502 14:14:07.324493 9553 tablet_copy_service.cc:161] P 5968a421103245be945b2453edc11c01: Beginning new tablet copy session on tablet cb6cacc3e4564116b13c696813ce0426 from peer d9e7b81687664b5f8009584f65b45847 at {username='slave'} at 127.6.196.200:55757: session id = d9e7b81687664b5f8009584f65b45847-cb6cacc3e4564116b13c696813ce0426
I20240502 14:14:07.326525 9553 tablet_copy_source_session.cc:205] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:07.327617 12697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb6cacc3e4564116b13c696813ce0426. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:07.331684 12697 tablet_copy_client.cc:783] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:07.331900 12697 tablet_copy_client.cc:647] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:07.333401 12697 tablet_copy_client.cc:523] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:07.335654 12697 tablet_bootstrap.cc:492] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847: Bootstrap starting.
I20240502 14:14:07.366495 12697 tablet_bootstrap.cc:492] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:07.366876 12697 tablet_bootstrap.cc:492] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847: Bootstrap complete.
I20240502 14:14:07.367121 12697 ts_tablet_manager.cc:1367] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847: Time spent bootstrapping tablet: real 0.032s user 0.027s sys 0.004s
I20240502 14:14:07.367864 12697 raft_consensus.cc:348] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: true } }
I20240502 14:14:07.368098 12697 raft_consensus.cc:729] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d9e7b81687664b5f8009584f65b45847, State: Initialized, Role: LEARNER
I20240502 14:14:07.368320 12697 consensus_queue.cc:260] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: true } }
I20240502 14:14:07.368898 12697 ts_tablet_manager.cc:1398] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:07.369707 9553 tablet_copy_service.cc:342] P 5968a421103245be945b2453edc11c01: Request end of tablet copy session d9e7b81687664b5f8009584f65b45847-cb6cacc3e4564116b13c696813ce0426 received from {username='slave'} at 127.6.196.200:55757
I20240502 14:14:07.369832 9553 tablet_copy_service.cc:433] P 5968a421103245be945b2453edc11c01: ending tablet copy session d9e7b81687664b5f8009584f65b45847-cb6cacc3e4564116b13c696813ce0426 on tablet cb6cacc3e4564116b13c696813ce0426 with peer d9e7b81687664b5f8009584f65b45847
I20240502 14:14:07.716095 12505 raft_consensus.cc:1203] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:07.837867 12505 raft_consensus.cc:1203] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:08.268936 12750 raft_consensus.cc:1050] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0: attempting to promote NON_VOTER d9e7b81687664b5f8009584f65b45847 to VOTER
I20240502 14:14:08.269515 12750 consensus_queue.cc:237] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } }
I20240502 14:14:08.271167 12505 raft_consensus.cc:1261] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [term 1 LEARNER]: Refusing update from remote peer 39deda1478724559830a8c119eae5be0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:08.271677 12750 consensus_queue.cc:1035] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:08.274847 12751 raft_consensus.cc:2932] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d9e7b81687664b5f8009584f65b45847 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } } }
I20240502 14:14:08.275496 12505 raft_consensus.cc:2932] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d9e7b81687664b5f8009584f65b45847 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } } }
I20240502 14:14:08.278954 9386 catalog_manager.cc:5484] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 reported cstate change: config changed from index 4 to 5, d9e7b81687664b5f8009584f65b45847 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "39deda1478724559830a8c119eae5be0" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:08.327225 12727 raft_consensus.cc:1050] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01: attempting to promote NON_VOTER d9e7b81687664b5f8009584f65b45847 to VOTER
I20240502 14:14:08.327812 12727 consensus_queue.cc:237] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } }
I20240502 14:14:08.329491 12505 raft_consensus.cc:1261] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [term 1 LEARNER]: Refusing update from remote peer 5968a421103245be945b2453edc11c01: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:08.330001 12727 consensus_queue.cc:1035] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:08.332270 12729 raft_consensus.cc:2932] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d9e7b81687664b5f8009584f65b45847 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } } }
I20240502 14:14:08.332809 12505 raft_consensus.cc:2932] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d9e7b81687664b5f8009584f65b45847 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } } }
I20240502 14:14:08.336572 9390 catalog_manager.cc:5484] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 reported cstate change: config changed from index 4 to 5, d9e7b81687664b5f8009584f65b45847 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5968a421103245be945b2453edc11c01" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:08.380723 9668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "828f6ad5260a469384cad06dd2986b05"
dest_uuid: "39deda1478724559830a8c119eae5be0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38558
I20240502 14:14:08.380940 9668 raft_consensus.cc:595] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:08.401197 9531 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cb6cacc3e4564116b13c696813ce0426"
dest_uuid: "5968a421103245be945b2453edc11c01"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48036
I20240502 14:14:08.401435 9531 raft_consensus.cc:595] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:08.618115 9668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "828f6ad5260a469384cad06dd2986b05"
dest_uuid: "39deda1478724559830a8c119eae5be0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38576
I20240502 14:14:08.618304 9668 raft_consensus.cc:595] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:08.633080 9531 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cb6cacc3e4564116b13c696813ce0426"
dest_uuid: "5968a421103245be945b2453edc11c01"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48056
I20240502 14:14:08.633301 9531 raft_consensus.cc:595] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:08.854768 9668 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "828f6ad5260a469384cad06dd2986b05"
dest_uuid: "39deda1478724559830a8c119eae5be0"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38606
I20240502 14:14:08.854966 9668 raft_consensus.cc:595] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:08.862599 12817 raft_consensus.cc:979] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0: : Instructing follower d9e7b81687664b5f8009584f65b45847 to start an election
I20240502 14:14:08.862730 12817 raft_consensus.cc:1067] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Signalling peer d9e7b81687664b5f8009584f65b45847 to start an election
I20240502 14:14:08.863427 12505 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "828f6ad5260a469384cad06dd2986b05"
dest_uuid: "d9e7b81687664b5f8009584f65b45847"
from {username='slave'} at 127.6.196.194:40143
I20240502 14:14:08.863660 12505 raft_consensus.cc:482] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:14:08.863852 12505 raft_consensus.cc:3037] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:08.866850 12505 raft_consensus.cc:504] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } }
I20240502 14:14:08.868127 12505 leader_election.cc:288] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [CANDIDATE]: Term 2 election: Requested vote from peers 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583)
I20240502 14:14:08.868526 9668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "828f6ad5260a469384cad06dd2986b05" candidate_uuid: "d9e7b81687664b5f8009584f65b45847" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "39deda1478724559830a8c119eae5be0"
I20240502 14:14:08.868727 9668 raft_consensus.cc:3032] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:14:08.868783 9668 raft_consensus.cc:729] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 39deda1478724559830a8c119eae5be0, State: Running, Role: LEADER
I20240502 14:14:08.868968 9668 consensus_queue.cc:260] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } }
I20240502 14:14:08.869167 9668 raft_consensus.cc:3037] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:08.870568 9531 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "cb6cacc3e4564116b13c696813ce0426"
dest_uuid: "5968a421103245be945b2453edc11c01"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48070
I20240502 14:14:08.870729 9531 raft_consensus.cc:595] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:14:08.871448 9668 raft_consensus.cc:2454] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9e7b81687664b5f8009584f65b45847 in term 2.
I20240502 14:14:08.872475 12438 leader_election.cc:302] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 39deda1478724559830a8c119eae5be0, d9e7b81687664b5f8009584f65b45847; no voters:
I20240502 14:14:08.872928 12823 raft_consensus.cc:2781] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:08.873214 12823 raft_consensus.cc:686] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [term 2 LEADER]: Becoming Leader. State: Replica: d9e7b81687664b5f8009584f65b45847, State: Running, Role: LEADER
I20240502 14:14:08.873656 12823 consensus_queue.cc:237] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } }
I20240502 14:14:08.876510 9386 catalog_manager.cc:5484] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 reported cstate change: term changed from 1 to 2, leader changed from 39deda1478724559830a8c119eae5be0 (127.6.196.194) to d9e7b81687664b5f8009584f65b45847 (127.6.196.200). New cstate: current_term: 2 leader_uuid: "d9e7b81687664b5f8009584f65b45847" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240502 14:14:08.885816 9374 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 828f6ad5260a469384cad06dd2986b05 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader d9e7b81687664b5f8009584f65b45847 (127.6.196.200:32911) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240502 14:14:08.927223 12831 raft_consensus.cc:979] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01: : Instructing follower d9e7b81687664b5f8009584f65b45847 to start an election
I20240502 14:14:08.927371 12831 raft_consensus.cc:1067] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Signalling peer d9e7b81687664b5f8009584f65b45847 to start an election
I20240502 14:14:08.927915 12505 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cb6cacc3e4564116b13c696813ce0426"
dest_uuid: "d9e7b81687664b5f8009584f65b45847"
from {username='slave'} at 127.6.196.193:60155
I20240502 14:14:08.928073 12505 raft_consensus.cc:482] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:14:08.928145 12505 raft_consensus.cc:3037] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:08.929837 12505 raft_consensus.cc:504] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } }
I20240502 14:14:08.930289 12505 leader_election.cc:288] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [CANDIDATE]: Term 2 election: Requested vote from peers 5968a421103245be945b2453edc11c01 (127.6.196.193:33389)
I20240502 14:14:08.930701 9531 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb6cacc3e4564116b13c696813ce0426" candidate_uuid: "d9e7b81687664b5f8009584f65b45847" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "5968a421103245be945b2453edc11c01"
I20240502 14:14:08.930862 9531 raft_consensus.cc:3032] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:14:08.930956 9531 raft_consensus.cc:729] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5968a421103245be945b2453edc11c01, State: Running, Role: LEADER
I20240502 14:14:08.931154 9531 consensus_queue.cc:260] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } }
I20240502 14:14:08.931380 9531 raft_consensus.cc:3037] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:14:08.932911 9531 raft_consensus.cc:2454] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d9e7b81687664b5f8009584f65b45847 in term 2.
I20240502 14:14:08.933272 12437 leader_election.cc:302] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5968a421103245be945b2453edc11c01, d9e7b81687664b5f8009584f65b45847; no voters:
I20240502 14:14:08.933483 12823 raft_consensus.cc:2781] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:14:08.933615 12823 raft_consensus.cc:686] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [term 2 LEADER]: Becoming Leader. State: Replica: d9e7b81687664b5f8009584f65b45847, State: Running, Role: LEADER
I20240502 14:14:08.933827 12823 consensus_queue.cc:237] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } }
I20240502 14:14:08.935549 9386 catalog_manager.cc:5484] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 reported cstate change: term changed from 1 to 2, leader changed from 5968a421103245be945b2453edc11c01 (127.6.196.193) to d9e7b81687664b5f8009584f65b45847 (127.6.196.200). New cstate: current_term: 2 leader_uuid: "d9e7b81687664b5f8009584f65b45847" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:09.310601 9668 raft_consensus.cc:1261] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Refusing update from remote peer d9e7b81687664b5f8009584f65b45847: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:14:09.311203 12823 consensus_queue.cc:1035] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "39deda1478724559830a8c119eae5be0" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 35583 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:09.388825 9531 raft_consensus.cc:1261] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Refusing update from remote peer d9e7b81687664b5f8009584f65b45847: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:14:09.389402 12823 consensus_queue.cc:1035] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5968a421103245be945b2453edc11c01" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 33389 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:09.495887 12504 consensus_queue.cc:237] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } }
I20240502 14:14:09.496662 12823 raft_consensus.cc:2932] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 39deda1478724559830a8c119eae5be0 (127.6.196.194) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } } }
I20240502 14:14:09.499346 9374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 828f6ad5260a469384cad06dd2986b05 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:14:09.500516 9390 catalog_manager.cc:5484] T 828f6ad5260a469384cad06dd2986b05 P d9e7b81687664b5f8009584f65b45847 reported cstate change: config changed from index 5 to 7, VOTER 39deda1478724559830a8c119eae5be0 (127.6.196.194) evicted. New cstate: current_term: 2 leader_uuid: "d9e7b81687664b5f8009584f65b45847" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:09.504040 9647 tablet_service.cc:1514] Processing DeleteTablet for tablet 828f6ad5260a469384cad06dd2986b05 with delete_type TABLET_DATA_TOMBSTONED (TS 39deda1478724559830a8c119eae5be0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:45240
I20240502 14:14:09.504671 12862 tablet_replica.cc:330] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0: stopping tablet replica
I20240502 14:14:09.504853 12862 raft_consensus.cc:2229] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:09.505096 12862 raft_consensus.cc:2258] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:09.506625 12862 ts_tablet_manager.cc:1875] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:09.510596 12862 ts_tablet_manager.cc:1888] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:14:09.510700 12862 log.cc:1196] T 828f6ad5260a469384cad06dd2986b05 P 39deda1478724559830a8c119eae5be0: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/828f6ad5260a469384cad06dd2986b05
I20240502 14:14:09.511330 9376 catalog_manager.cc:4830] TS 39deda1478724559830a8c119eae5be0 (127.6.196.194:35583): tablet 828f6ad5260a469384cad06dd2986b05 (table rebalance_test_table_2 [id=1af730f9592947898d4b5e6f2b642194]) successfully deleted
I20240502 14:14:09.547500 12504 consensus_queue.cc:237] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } }
I20240502 14:14:09.548221 12823 raft_consensus.cc:2932] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 5968a421103245be945b2453edc11c01 (127.6.196.193) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } } }
I20240502 14:14:09.550909 9374 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cb6cacc3e4564116b13c696813ce0426 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240502 14:14:09.551997 9390 catalog_manager.cc:5484] T cb6cacc3e4564116b13c696813ce0426 P d9e7b81687664b5f8009584f65b45847 reported cstate change: config changed from index 5 to 7, VOTER 5968a421103245be945b2453edc11c01 (127.6.196.193) evicted. New cstate: current_term: 2 leader_uuid: "d9e7b81687664b5f8009584f65b45847" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "d9e7b81687664b5f8009584f65b45847" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 32911 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:09.554651 9511 tablet_service.cc:1514] Processing DeleteTablet for tablet cb6cacc3e4564116b13c696813ce0426 with delete_type TABLET_DATA_TOMBSTONED (TS 5968a421103245be945b2453edc11c01 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56970
I20240502 14:14:09.555205 12878 tablet_replica.cc:330] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01: stopping tablet replica
I20240502 14:14:09.555356 12878 raft_consensus.cc:2229] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:09.555505 12878 raft_consensus.cc:2258] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:09.556864 12878 ts_tablet_manager.cc:1875] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:09.560593 12878 ts_tablet_manager.cc:1888] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:14:09.560680 12878 log.cc:1196] T cb6cacc3e4564116b13c696813ce0426 P 5968a421103245be945b2453edc11c01: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/cb6cacc3e4564116b13c696813ce0426
I20240502 14:14:09.561355 9375 catalog_manager.cc:4830] TS 5968a421103245be945b2453edc11c01 (127.6.196.193:33389): tablet cb6cacc3e4564116b13c696813ce0426 (table rebalance_test_table_3 [id=21a0de7bef9c4ac685e4c4a9286573eb]) successfully deleted
I20240502 14:14:09.826704 10762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.829387 10898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.847622 9647 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.856869 9511 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.866374 11030 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.893411 11308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.908293 12485 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:09.920766 11163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.055042 11030 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.063117 10762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.064697 10898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.087589 9647 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.108484 9511 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.109589 11163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.130028 11308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.155030 12485 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.391558 10762 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.400779 10898 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.401187 11030 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.410262 9647 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.420671 9511 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.434054 11163 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.453060 11308 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:10.471349 12485 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
247adb9a8de14791b6b2872e02aeff89 | 127.6.196.254:43585 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.6.196.212:43875 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
2c6030342d9b40bd91c198e3634a041a | 127.6.196.195:41811 | HEALTHY | <none> | 2 | 0
2f7a0e54a60741298e73367e8bbf1237 | 127.6.196.197:45913 | HEALTHY | <none> | 2 | 0
33de03b632d64afe8eca2f78b08fa42e | 127.6.196.196:32821 | HEALTHY | <none> | 2 | 0
39deda1478724559830a8c119eae5be0 | 127.6.196.194:35583 | HEALTHY | <none> | 2 | 0
5968a421103245be945b2453edc11c01 | 127.6.196.193:33389 | HEALTHY | <none> | 2 | 0
99b8f6d3810b4240ba87893565a08ce9 | 127.6.196.198:34211 | HEALTHY | <none> | 2 | 0
b4b9d2cad7e04837bc20566099917739 | 127.6.196.199:43261 | HEALTHY | <none> | 2 | 0
d9e7b81687664b5f8009584f65b45847 | 127.6.196.200:32911 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.6.196.193 | experimental | 127.6.196.193:33389
local_ip_for_outbound_sockets | 127.6.196.194 | experimental | 127.6.196.194:35583
local_ip_for_outbound_sockets | 127.6.196.195 | experimental | 127.6.196.195:41811
local_ip_for_outbound_sockets | 127.6.196.196 | experimental | 127.6.196.196:32821
local_ip_for_outbound_sockets | 127.6.196.197 | experimental | 127.6.196.197:45913
local_ip_for_outbound_sockets | 127.6.196.198 | experimental | 127.6.196.198:34211
local_ip_for_outbound_sockets | 127.6.196.199 | experimental | 127.6.196.199:43261
local_ip_for_outbound_sockets | 127.6.196.200 | experimental | 127.6.196.200:32911
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.6.196.193:33389
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.6.196.194:35583
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.6.196.195:41811
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.6.196.196:32821
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.6.196.197:45913
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.6.196.198:34211
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.6.196.199:43261
server_dump_info_path | /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.6.196.200:32911
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.6.196.212:43875 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240502 14:14:10.514132 6931 log_verifier.cc:126] Checking tablet 40c7c0b7f94b42b392afd30752d26196
I20240502 14:14:10.523597 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 40c7c0b7f94b42b392afd30752d26196
I20240502 14:14:10.523694 6931 log_verifier.cc:126] Checking tablet 4ae2df4e083e4205a815a77897710ab1
I20240502 14:14:10.532882 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 4ae2df4e083e4205a815a77897710ab1
I20240502 14:14:10.532963 6931 log_verifier.cc:126] Checking tablet 594efe32e19c46658945080d250ca29c
I20240502 14:14:10.541275 6931 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 594efe32e19c46658945080d250ca29c
I20240502 14:14:10.541359 6931 log_verifier.cc:126] Checking tablet 5f8493b9da2c4cf29de9a1ac4c70cac0
I20240502 14:14:10.546093 6931 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 5f8493b9da2c4cf29de9a1ac4c70cac0
I20240502 14:14:10.546152 6931 log_verifier.cc:126] Checking tablet 63746cf04e6145749291b0c9e28b4b82
I20240502 14:14:10.555745 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 63746cf04e6145749291b0c9e28b4b82
I20240502 14:14:10.555825 6931 log_verifier.cc:126] Checking tablet 6adbf15fbe4f49c5b10a30fd608e058a
I20240502 14:14:10.564921 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6adbf15fbe4f49c5b10a30fd608e058a
I20240502 14:14:10.564986 6931 log_verifier.cc:126] Checking tablet 7d4a89cb247c4b8ea952c245ca6f6377
I20240502 14:14:10.573318 6931 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7d4a89cb247c4b8ea952c245ca6f6377
I20240502 14:14:10.573385 6931 log_verifier.cc:126] Checking tablet 828f6ad5260a469384cad06dd2986b05
I20240502 14:14:10.584808 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 828f6ad5260a469384cad06dd2986b05
I20240502 14:14:10.584878 6931 log_verifier.cc:126] Checking tablet 82e72d3de4e34731b7044de67bf6d635
I20240502 14:14:10.593847 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 82e72d3de4e34731b7044de67bf6d635
I20240502 14:14:10.593904 6931 log_verifier.cc:126] Checking tablet 93d3db61bd8143d48d66ce379e443323
I20240502 14:14:10.602953 6931 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 93d3db61bd8143d48d66ce379e443323
I20240502 14:14:10.603014 6931 log_verifier.cc:126] Checking tablet 9a2af91313e0451391e05d10dce3c2ea
I20240502 14:14:10.616191 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9a2af91313e0451391e05d10dce3c2ea
I20240502 14:14:10.616269 6931 log_verifier.cc:126] Checking tablet c8aba452497f40ec971f5784d3edbfd1
I20240502 14:14:10.628226 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c8aba452497f40ec971f5784d3edbfd1
I20240502 14:14:10.628291 6931 log_verifier.cc:126] Checking tablet cb6cacc3e4564116b13c696813ce0426
I20240502 14:14:10.639568 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet cb6cacc3e4564116b13c696813ce0426
I20240502 14:14:10.639642 6931 log_verifier.cc:126] Checking tablet cf42265c4952414d90b9ac9d8fb934e6
I20240502 14:14:10.648044 6931 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cf42265c4952414d90b9ac9d8fb934e6
I20240502 14:14:10.648120 6931 log_verifier.cc:126] Checking tablet db4824dd6aa44ad7bd2890238e006975
I20240502 14:14:10.656958 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet db4824dd6aa44ad7bd2890238e006975
I20240502 14:14:10.657020 6931 log_verifier.cc:126] Checking tablet ffe9dcb04b354f61a43df073c78aec59
I20240502 14:14:10.665693 6931 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ffe9dcb04b354f61a43df073c78aec59
I20240502 14:14:10.666497 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 9449
I20240502 14:14:10.667760 9572 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:10.795182 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 9449
I20240502 14:14:10.820344 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 9583
I20240502 14:14:10.821350 9709 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:10.954965 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 9583
I20240502 14:14:10.978511 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 10698
I20240502 14:14:10.984393 10825 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:11.092060 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 10698
I20240502 14:14:11.112469 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 10835
I20240502 14:14:11.118242 10959 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:11.223893 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 10835
I20240502 14:14:11.244119 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 10967
I20240502 14:14:11.249819 11089 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:11.359853 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 10967
W20240502 14:14:11.379484 9377 connection.cc:535] server connection from 127.6.196.197:33489 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:14:11.379742 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 11099
I20240502 14:14:11.384855 11226 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:11.489073 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 11099
I20240502 14:14:11.508349 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 11236
I20240502 14:14:11.513464 11367 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:11.618093 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 11236
I20240502 14:14:11.637526 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 12421
I20240502 14:14:11.643069 12544 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:11.736070 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 12421
I20240502 14:14:11.754163 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 9358
I20240502 14:14:11.755142 9420 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:11.890257 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 9358
2024-05-02T14:14:11Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (26231 ms)
[----------] 1 test from RebalanceParamTest (26231 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240502 14:14:11.953442 6931 test_util.cc:268] Using random seed: 933207824
I20240502 14:14:11.955049 6931 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:14:11.955103 6931 ts_itest-base.cc:113] --------------
I20240502 14:14:11.955152 6931 ts_itest-base.cc:114] 8 tablet servers
I20240502 14:14:11.955200 6931 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:14:11.955265 6931 ts_itest-base.cc:116] --------------
2024-05-02T14:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:14:11Z Disabled control of system clock
I20240502 14:14:11.981518 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.196.254:44303
--webserver_interface=127.6.196.254
--webserver_port=0
--builtin_ntp_servers=127.6.196.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.196.254:44303
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 14:14:12.274775 13014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:12.275080 13014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:12.275213 13014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:12.281829 13014 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:12.281941 13014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:14:12.282011 13014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:12.282088 13014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:14:12.282150 13014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:14:12.290020 13014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.196.254:44303
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.196.254:44303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.196.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:12.291080 13014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:12.292562 13014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:12.302693 13019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:12.303231 13020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:12.304030 13022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:12.304642 13014 server_base.cc:1034] running on GCE node
I20240502 14:14:12.305405 13014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:12.306576 13014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:12.307766 13014 hybrid_clock.cc:648] HybridClock initialized: now 1714659252307719 us; error 41 us; skew 500 ppm
I20240502 14:14:12.308055 13014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:12.310093 13014 webserver.cc:460] Webserver started at http://127.6.196.254:42739/ using document root <none> and password file <none>
I20240502 14:14:12.310688 13014 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:12.310798 13014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:12.311067 13014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:12.313412 13014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f161cc9c91884c50874ece8743051e73"
format_stamp: "Formatted at 2024-05-02 14:14:12 on dist-test-slave-kjf7"
I20240502 14:14:12.314095 13014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f161cc9c91884c50874ece8743051e73"
format_stamp: "Formatted at 2024-05-02 14:14:12 on dist-test-slave-kjf7"
I20240502 14:14:12.318604 13014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:14:12.321540 13028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:12.322662 13014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:14:12.322818 13014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f161cc9c91884c50874ece8743051e73"
format_stamp: "Formatted at 2024-05-02 14:14:12 on dist-test-slave-kjf7"
I20240502 14:14:12.322983 13014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:12.347410 13014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:12.348078 13014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:12.348320 13014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:12.364280 13014 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.254:44303
I20240502 14:14:12.364339 13080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.254:44303 every 8 connection(s)
I20240502 14:14:12.365505 13014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:14:12.368790 13081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:12.372105 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 13014
I20240502 14:14:12.372309 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:14:12.378901 13081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Bootstrap starting.
I20240502 14:14:12.382212 13081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:12.383293 13081 log.cc:824] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:12.386186 13081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: No bootstrap required, opened a new log
I20240502 14:14:12.391904 13081 raft_consensus.cc:348] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f161cc9c91884c50874ece8743051e73" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 44303 } }
I20240502 14:14:12.392164 13081 raft_consensus.cc:374] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:12.392230 13081 raft_consensus.cc:729] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f161cc9c91884c50874ece8743051e73, State: Initialized, Role: FOLLOWER
I20240502 14:14:12.392673 13081 consensus_queue.cc:260] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f161cc9c91884c50874ece8743051e73" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 44303 } }
I20240502 14:14:12.392850 13081 raft_consensus.cc:388] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:14:12.392942 13081 raft_consensus.cc:482] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:14:12.393051 13081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:12.395020 13081 raft_consensus.cc:504] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f161cc9c91884c50874ece8743051e73" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 44303 } }
I20240502 14:14:12.395589 13081 leader_election.cc:302] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f161cc9c91884c50874ece8743051e73; no voters:
I20240502 14:14:12.396167 13081 leader_election.cc:288] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:14:12.396268 13086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:12.396831 13086 raft_consensus.cc:686] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [term 1 LEADER]: Becoming Leader. State: Replica: f161cc9c91884c50874ece8743051e73, State: Running, Role: LEADER
I20240502 14:14:12.397361 13086 consensus_queue.cc:237] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f161cc9c91884c50874ece8743051e73" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 44303 } }
I20240502 14:14:12.397933 13081 sys_catalog.cc:564] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:14:12.400077 13087 sys_catalog.cc:455] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f161cc9c91884c50874ece8743051e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f161cc9c91884c50874ece8743051e73" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 44303 } } }
I20240502 14:14:12.400344 13087 sys_catalog.cc:458] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:12.400157 13088 sys_catalog.cc:455] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f161cc9c91884c50874ece8743051e73. Latest consensus state: current_term: 1 leader_uuid: "f161cc9c91884c50874ece8743051e73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f161cc9c91884c50874ece8743051e73" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 44303 } } }
I20240502 14:14:12.400480 13088 sys_catalog.cc:458] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73 [sys.catalog]: This master's current role is: LEADER
I20240502 14:14:12.401984 13091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:14:12.406461 13091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:14:12.415596 13091 catalog_manager.cc:1347] Generated new cluster ID: e0e88d1843b04640bcb2f57418c61860
I20240502 14:14:12.415726 13091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:14:12.431870 13091 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:14:12.433188 13091 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:14:12.441637 13091 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Generated new TSK 0
I20240502 14:14:12.442318 13091 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:14:12.456533 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.193:0
--local_ip_for_outbound_sockets=127.6.196.193
--webserver_interface=127.6.196.193
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--builtin_ntp_servers=127.6.196.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:14:12.756523 13105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:12.756759 13105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:12.756835 13105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:12.763504 13105 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:12.763594 13105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:12.763697 13105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.193
I20240502 14:14:12.771392 13105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.196.193
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.193
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:12.772410 13105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:12.773872 13105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:12.784598 13110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:12.785421 13113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:12.786033 13111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:12.786625 13105 server_base.cc:1034] running on GCE node
I20240502 14:14:12.787115 13105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:12.787895 13105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:12.789083 13105 hybrid_clock.cc:648] HybridClock initialized: now 1714659252789050 us; error 51 us; skew 500 ppm
I20240502 14:14:12.789376 13105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:12.791196 13105 webserver.cc:460] Webserver started at http://127.6.196.193:42557/ using document root <none> and password file <none>
I20240502 14:14:12.791785 13105 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:12.791874 13105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:12.792136 13105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:12.794236 13105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "20a967bad5504b1e9c6fb720ecdd1625"
format_stamp: "Formatted at 2024-05-02 14:14:12 on dist-test-slave-kjf7"
I20240502 14:14:12.794914 13105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "20a967bad5504b1e9c6fb720ecdd1625"
format_stamp: "Formatted at 2024-05-02 14:14:12 on dist-test-slave-kjf7"
I20240502 14:14:12.799420 13105 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:14:12.802160 13119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:12.803177 13105 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:12.803303 13105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "20a967bad5504b1e9c6fb720ecdd1625"
format_stamp: "Formatted at 2024-05-02 14:14:12 on dist-test-slave-kjf7"
I20240502 14:14:12.803457 13105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:12.819394 13105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:12.820041 13105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:12.820256 13105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:12.821075 13105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:12.822290 13105 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:12.822364 13105 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:12.822458 13105 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:12.822528 13105 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:12.851366 13105 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.193:36687
I20240502 14:14:12.851383 13232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.193:36687 every 8 connection(s)
I20240502 14:14:12.852406 13105 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:14:12.857868 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 13105
I20240502 14:14:12.858063 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:14:12.861073 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.194:0
--local_ip_for_outbound_sockets=127.6.196.194
--webserver_interface=127.6.196.194
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--builtin_ntp_servers=127.6.196.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:12.863410 13233 heartbeater.cc:347] Connected to a master server at 127.6.196.254:44303
I20240502 14:14:12.863672 13233 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:12.864478 13233 heartbeater.cc:511] Master 127.6.196.254:44303 requested a full tablet report, sending...
I20240502 14:14:12.866557 13046 ts_manager.cc:194] Registered new tserver with Master: 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:12.868467 13046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.193:34259
W20240502 14:14:13.166903 13236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:13.167184 13236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:13.167270 13236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:13.173977 13236 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:13.174077 13236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:13.174234 13236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.194
I20240502 14:14:13.182236 13236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.196.194
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.194
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:13.183311 13236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:13.184844 13236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:13.195928 13242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:13.196504 13241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:13.196959 13244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:13.197947 13236 server_base.cc:1034] running on GCE node
I20240502 14:14:13.198489 13236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:13.199393 13236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:13.200604 13236 hybrid_clock.cc:648] HybridClock initialized: now 1714659253200557 us; error 48 us; skew 500 ppm
I20240502 14:14:13.200865 13236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:13.202705 13236 webserver.cc:460] Webserver started at http://127.6.196.194:43505/ using document root <none> and password file <none>
I20240502 14:14:13.203301 13236 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:13.203399 13236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:13.203653 13236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:13.205976 13236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "cd67493f6f0142f896010f2b8a39887d"
format_stamp: "Formatted at 2024-05-02 14:14:13 on dist-test-slave-kjf7"
I20240502 14:14:13.206704 13236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "cd67493f6f0142f896010f2b8a39887d"
format_stamp: "Formatted at 2024-05-02 14:14:13 on dist-test-slave-kjf7"
I20240502 14:14:13.211297 13236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:14:13.214218 13250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:13.215312 13236 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240502 14:14:13.215456 13236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cd67493f6f0142f896010f2b8a39887d"
format_stamp: "Formatted at 2024-05-02 14:14:13 on dist-test-slave-kjf7"
I20240502 14:14:13.215631 13236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:13.233783 13236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:13.234436 13236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:13.234663 13236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:13.235500 13236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:13.236793 13236 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:13.236879 13236 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:13.236979 13236 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:13.237044 13236 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:13.265347 13236 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.194:33193
I20240502 14:14:13.265404 13363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.194:33193 every 8 connection(s)
I20240502 14:14:13.266494 13236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:14:13.270455 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 13236
I20240502 14:14:13.270592 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:14:13.273202 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.195:0
--local_ip_for_outbound_sockets=127.6.196.195
--webserver_interface=127.6.196.195
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--builtin_ntp_servers=127.6.196.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:13.277437 13364 heartbeater.cc:347] Connected to a master server at 127.6.196.254:44303
I20240502 14:14:13.277676 13364 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:13.278339 13364 heartbeater.cc:511] Master 127.6.196.254:44303 requested a full tablet report, sending...
I20240502 14:14:13.279563 13046 ts_manager.cc:194] Registered new tserver with Master: cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:13.280282 13046 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.194:59215
W20240502 14:14:13.567021 13367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:13.567238 13367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:13.567364 13367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:13.573874 13367 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:13.573972 13367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:13.574123 13367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.195
I20240502 14:14:13.582063 13367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.196.195
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.195
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:13.583061 13367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:13.584550 13367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:13.595932 13372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:13.596635 13373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:13.597362 13375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:13.598050 13367 server_base.cc:1034] running on GCE node
I20240502 14:14:13.598575 13367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:13.599383 13367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:13.600580 13367 hybrid_clock.cc:648] HybridClock initialized: now 1714659253600537 us; error 38 us; skew 500 ppm
I20240502 14:14:13.600862 13367 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:13.602922 13367 webserver.cc:460] Webserver started at http://127.6.196.195:41989/ using document root <none> and password file <none>
I20240502 14:14:13.603551 13367 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:13.603657 13367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:13.603971 13367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:13.606372 13367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "88b77f6edc1c4a72b548bf47c05704f3"
format_stamp: "Formatted at 2024-05-02 14:14:13 on dist-test-slave-kjf7"
I20240502 14:14:13.607154 13367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "88b77f6edc1c4a72b548bf47c05704f3"
format_stamp: "Formatted at 2024-05-02 14:14:13 on dist-test-slave-kjf7"
I20240502 14:14:13.611850 13367 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:14:13.614797 13381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:13.615937 13367 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:13.616093 13367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "88b77f6edc1c4a72b548bf47c05704f3"
format_stamp: "Formatted at 2024-05-02 14:14:13 on dist-test-slave-kjf7"
I20240502 14:14:13.616259 13367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:13.628746 13367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:13.629468 13367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:13.629709 13367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:13.630558 13367 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:13.631875 13367 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:13.631968 13367 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:13.632073 13367 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:13.632148 13367 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:13.660488 13367 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.195:46303
I20240502 14:14:13.660528 13494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.195:46303 every 8 connection(s)
I20240502 14:14:13.661803 13367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:14:13.671314 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 13367
I20240502 14:14:13.671480 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:14:13.673188 13495 heartbeater.cc:347] Connected to a master server at 127.6.196.254:44303
I20240502 14:14:13.673462 13495 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:13.674273 13495 heartbeater.cc:511] Master 127.6.196.254:44303 requested a full tablet report, sending...
I20240502 14:14:13.674728 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.196:0
--local_ip_for_outbound_sockets=127.6.196.196
--webserver_interface=127.6.196.196
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--builtin_ntp_servers=127.6.196.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:13.675621 13045 ts_manager.cc:194] Registered new tserver with Master: 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:13.676383 13045 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.195:57155
I20240502 14:14:13.871408 13233 heartbeater.cc:503] Master 127.6.196.254:44303 was elected leader, sending a full tablet report...
W20240502 14:14:13.979488 13498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:13.979727 13498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:13.979835 13498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:13.986363 13498 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:13.986454 13498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:13.986585 13498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.196
I20240502 14:14:13.993834 13498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.196.196
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.196
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:13.994868 13498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:13.996470 13498 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:14.007249 13503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:14.007779 13504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:14.008533 13506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:14.009246 13498 server_base.cc:1034] running on GCE node
I20240502 14:14:14.009737 13498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:14.010622 13498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:14.011812 13498 hybrid_clock.cc:648] HybridClock initialized: now 1714659254011779 us; error 30 us; skew 500 ppm
I20240502 14:14:14.012068 13498 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:14.019084 13498 webserver.cc:460] Webserver started at http://127.6.196.196:36191/ using document root <none> and password file <none>
I20240502 14:14:14.020079 13498 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:14.020220 13498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:14.020582 13498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:14.023304 13498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "7eb4ecb2379b44c0954f2658c4a04350"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-kjf7"
I20240502 14:14:14.023953 13498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "7eb4ecb2379b44c0954f2658c4a04350"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-kjf7"
I20240502 14:14:14.028587 13498 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20240502 14:14:14.031531 13513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.032722 13498 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:14.032847 13498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "7eb4ecb2379b44c0954f2658c4a04350"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-kjf7"
I20240502 14:14:14.033004 13498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:14.079232 13498 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:14.080040 13498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:14.080300 13498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:14.081175 13498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:14.082497 13498 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:14.082572 13498 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.082672 13498 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:14.082727 13498 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.110152 13498 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.196:46817
I20240502 14:14:14.110227 13625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.196:46817 every 8 connection(s)
I20240502 14:14:14.111351 13498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:14:14.113536 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 13498
I20240502 14:14:14.113704 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:14:14.116539 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.197:0
--local_ip_for_outbound_sockets=127.6.196.197
--webserver_interface=127.6.196.197
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--builtin_ntp_servers=127.6.196.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:14.123046 13626 heartbeater.cc:347] Connected to a master server at 127.6.196.254:44303
I20240502 14:14:14.123312 13626 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:14.124204 13626 heartbeater.cc:511] Master 127.6.196.254:44303 requested a full tablet report, sending...
I20240502 14:14:14.125644 13045 ts_manager.cc:194] Registered new tserver with Master: 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:14.126422 13045 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.196:52165
I20240502 14:14:14.282236 13364 heartbeater.cc:503] Master 127.6.196.254:44303 was elected leader, sending a full tablet report...
W20240502 14:14:14.411499 13629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:14.411751 13629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:14.411883 13629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:14.418658 13629 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:14.418772 13629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:14.418902 13629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.197
I20240502 14:14:14.427138 13629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.196.197
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.197
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:14.428217 13629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:14.429720 13629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:14.440717 13636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:14.441459 13635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:14.442049 13638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:14.442703 13629 server_base.cc:1034] running on GCE node
I20240502 14:14:14.443181 13629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:14.443910 13629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:14.445094 13629 hybrid_clock.cc:648] HybridClock initialized: now 1714659254445050 us; error 34 us; skew 500 ppm
I20240502 14:14:14.445402 13629 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:14.447361 13629 webserver.cc:460] Webserver started at http://127.6.196.197:34095/ using document root <none> and password file <none>
I20240502 14:14:14.448009 13629 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:14.448109 13629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:14.448371 13629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:14.450557 13629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "d1da7f11f0604f78934354a992bb0f55"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-kjf7"
I20240502 14:14:14.451220 13629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "d1da7f11f0604f78934354a992bb0f55"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-kjf7"
I20240502 14:14:14.455619 13629 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:14:14.458551 13644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.459645 13629 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:14.459779 13629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d1da7f11f0604f78934354a992bb0f55"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-kjf7"
I20240502 14:14:14.459975 13629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:14.480696 13629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:14.481356 13629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:14.481591 13629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:14.482422 13629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:14.483615 13629 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:14.483702 13629 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.483803 13629 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:14.483870 13629 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.512291 13629 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.197:43695
I20240502 14:14:14.512341 13759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.197:43695 every 8 connection(s)
I20240502 14:14:14.513361 13629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:14:14.516014 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 13629
I20240502 14:14:14.516186 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:14:14.519187 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.198:0
--local_ip_for_outbound_sockets=127.6.196.198
--webserver_interface=127.6.196.198
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--builtin_ntp_servers=127.6.196.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:14.525252 13760 heartbeater.cc:347] Connected to a master server at 127.6.196.254:44303
I20240502 14:14:14.525563 13760 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:14.526461 13760 heartbeater.cc:511] Master 127.6.196.254:44303 requested a full tablet report, sending...
I20240502 14:14:14.527874 13045 ts_manager.cc:194] Registered new tserver with Master: d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:14.528554 13045 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.197:37919
I20240502 14:14:14.678885 13495 heartbeater.cc:503] Master 127.6.196.254:44303 was elected leader, sending a full tablet report...
W20240502 14:14:14.810343 13763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:14.810566 13763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:14.810686 13763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:14.816820 13763 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:14.816922 13763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:14.817054 13763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.198
I20240502 14:14:14.824409 13763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.196.198
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.198
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:14.825430 13763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:14.826906 13763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:14.837209 13769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:14.837738 13768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:14.838503 13771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:14.839172 13763 server_base.cc:1034] running on GCE node
I20240502 14:14:14.839586 13763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:14.840304 13763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:14.841468 13763 hybrid_clock.cc:648] HybridClock initialized: now 1714659254841426 us; error 38 us; skew 500 ppm
I20240502 14:14:14.841707 13763 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:14.843425 13763 webserver.cc:460] Webserver started at http://127.6.196.198:33425/ using document root <none> and password file <none>
I20240502 14:14:14.843991 13763 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:14.844087 13763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:14.844341 13763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:14.846613 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "c86f7440570f4796babfd44c3d7fd847"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-kjf7"
I20240502 14:14:14.847252 13763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "c86f7440570f4796babfd44c3d7fd847"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-kjf7"
I20240502 14:14:14.851475 13763 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 14:14:14.854784 13777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.855887 13763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:14.856034 13763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c86f7440570f4796babfd44c3d7fd847"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-kjf7"
I20240502 14:14:14.856215 13763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:14.881368 13763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:14.882033 13763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:14.882269 13763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:14.883070 13763 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:14.884287 13763 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:14.884373 13763 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.884477 13763 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:14.884547 13763 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:14.911963 13763 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.198:42041
I20240502 14:14:14.912022 13892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.198:42041 every 8 connection(s)
I20240502 14:14:14.912946 13763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:14:14.917995 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 13763
I20240502 14:14:14.918134 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:14:14.921090 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.199:0
--local_ip_for_outbound_sockets=127.6.196.199
--webserver_interface=127.6.196.199
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--builtin_ntp_servers=127.6.196.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:14.923902 13893 heartbeater.cc:347] Connected to a master server at 127.6.196.254:44303
I20240502 14:14:14.924153 13893 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:14.924777 13893 heartbeater.cc:511] Master 127.6.196.254:44303 requested a full tablet report, sending...
I20240502 14:14:14.926162 13045 ts_manager.cc:194] Registered new tserver with Master: c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:14.927131 13045 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.198:36111
I20240502 14:14:15.128986 13626 heartbeater.cc:503] Master 127.6.196.254:44303 was elected leader, sending a full tablet report...
W20240502 14:14:15.229002 13897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:15.229274 13897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:15.229400 13897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:15.235880 13897 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:15.236001 13897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:15.236150 13897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.199
I20240502 14:14:15.244060 13897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.196.199
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.199
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:15.245182 13897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:15.246848 13897 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:15.258028 13903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:15.258736 13902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:15.259339 13905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:15.259914 13897 server_base.cc:1034] running on GCE node
I20240502 14:14:15.260468 13897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:15.261363 13897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:15.262542 13897 hybrid_clock.cc:648] HybridClock initialized: now 1714659255262501 us; error 37 us; skew 500 ppm
I20240502 14:14:15.262789 13897 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:15.264822 13897 webserver.cc:460] Webserver started at http://127.6.196.199:35701/ using document root <none> and password file <none>
I20240502 14:14:15.265740 13897 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:15.265861 13897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:15.266245 13897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:15.269166 13897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "5297e816f782430d9a01220f0471a882"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-kjf7"
I20240502 14:14:15.269850 13897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "5297e816f782430d9a01220f0471a882"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-kjf7"
I20240502 14:14:15.274320 13897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:14:15.277213 13911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:15.278400 13897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:14:15.278522 13897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5297e816f782430d9a01220f0471a882"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-kjf7"
I20240502 14:14:15.278673 13897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:15.292306 13897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:15.292941 13897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:15.293179 13897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:15.293991 13897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:15.295203 13897 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:15.295284 13897 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:15.295358 13897 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:15.295411 13897 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:15.323300 13897 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.199:33857
I20240502 14:14:15.323348 14024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.199:33857 every 8 connection(s)
I20240502 14:14:15.324487 13897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:14:15.329809 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 13897
I20240502 14:14:15.330039 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:14:15.332984 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.200:0
--local_ip_for_outbound_sockets=127.6.196.200
--webserver_interface=127.6.196.200
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--builtin_ntp_servers=127.6.196.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:15.336620 14025 heartbeater.cc:347] Connected to a master server at 127.6.196.254:44303
I20240502 14:14:15.336906 14025 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:15.337841 14025 heartbeater.cc:511] Master 127.6.196.254:44303 requested a full tablet report, sending...
I20240502 14:14:15.339207 13045 ts_manager.cc:194] Registered new tserver with Master: 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:14:15.340010 13045 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.199:37449
I20240502 14:14:15.531096 13760 heartbeater.cc:503] Master 127.6.196.254:44303 was elected leader, sending a full tablet report...
W20240502 14:14:15.627326 14028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:15.627600 14028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:15.627725 14028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:15.634025 14028 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:15.634133 14028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:15.634263 14028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.200
I20240502 14:14:15.642114 14028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.196.200
--webserver_port=0
--tserver_master_addrs=127.6.196.254:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.200
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:15.643174 14028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:15.644785 14028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:15.655938 14033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:15.656420 14034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:15.657334 14036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:15.658000 14028 server_base.cc:1034] running on GCE node
I20240502 14:14:15.658463 14028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:15.659260 14028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:15.660444 14028 hybrid_clock.cc:648] HybridClock initialized: now 1714659255660387 us; error 44 us; skew 500 ppm
I20240502 14:14:15.660755 14028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:15.662688 14028 webserver.cc:460] Webserver started at http://127.6.196.200:44151/ using document root <none> and password file <none>
I20240502 14:14:15.663290 14028 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:15.663374 14028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:15.663594 14028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:14:15.665843 14028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "8b47d4390f8343a493d6c00123785968"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-kjf7"
I20240502 14:14:15.666502 14028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "8b47d4390f8343a493d6c00123785968"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-kjf7"
I20240502 14:14:15.670559 14028 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:14:15.673311 14042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:15.674329 14028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:14:15.674439 14028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "8b47d4390f8343a493d6c00123785968"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-kjf7"
I20240502 14:14:15.674573 14028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:15.700067 14028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:15.700673 14028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:15.700912 14028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:15.701722 14028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:15.702888 14028 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:15.702961 14028 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:15.703028 14028 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:15.703075 14028 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:15.738905 14028 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.200:43857
I20240502 14:14:15.738971 14160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.200:43857 every 8 connection(s)
I20240502 14:14:15.740093 14028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:14:15.742314 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 14028
I20240502 14:14:15.742465 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:14:15.759230 14161 heartbeater.cc:347] Connected to a master server at 127.6.196.254:44303
I20240502 14:14:15.759644 14161 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:15.760603 14161 heartbeater.cc:511] Master 127.6.196.254:44303 requested a full tablet report, sending...
I20240502 14:14:15.762046 13045 ts_manager.cc:194] Registered new tserver with Master: 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:15.763238 13045 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.200:52219
I20240502 14:14:15.771780 6931 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:14:15.787703 6931 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:14:15.787822 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 13629
I20240502 14:14:15.795464 13755 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:15.869378 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 13629
I20240502 14:14:15.884765 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 13763
I20240502 14:14:15.892057 13888 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:15.964468 13893 heartbeater.cc:503] Master 127.6.196.254:44303 was elected leader, sending a full tablet report...
I20240502 14:14:15.965553 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 13763
W20240502 14:14:15.981490 13029 connection.cc:535] server connection from 127.6.196.198:36111 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:14:15.981671 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 13897
I20240502 14:14:15.988786 14020 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:16.063028 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 13897
I20240502 14:14:16.077528 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 14028
I20240502 14:14:16.085268 14156 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:14:16.160207 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 14028
I20240502 14:14:19.927711 13045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:19.961333 13167 tablet_service.cc:1467] Processing CreateTablet for tablet 257d72583cd94c0d8efdd8925be48dca (DEFAULT_TABLE table=rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:19.961179 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 257d72583cd94c0d8efdd8925be48dca (DEFAULT_TABLE table=rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:19.961194 13168 tablet_service.cc:1467] Processing CreateTablet for tablet fd031f1ac25b42769006740ba81a96b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:19.962601 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 257d72583cd94c0d8efdd8925be48dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:19.962615 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd031f1ac25b42769006740ba81a96b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:19.961506 13166 tablet_service.cc:1467] Processing CreateTablet for tablet a5969155f010461293a7f2f61c88c817 (DEFAULT_TABLE table=rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:19.962997 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5969155f010461293a7f2f61c88c817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:19.962937 13298 tablet_service.cc:1467] Processing CreateTablet for tablet fd031f1ac25b42769006740ba81a96b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:19.964154 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd031f1ac25b42769006740ba81a96b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:19.964429 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 257d72583cd94c0d8efdd8925be48dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:19.968561 13561 tablet_service.cc:1467] Processing CreateTablet for tablet fd031f1ac25b42769006740ba81a96b9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:19.970379 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd031f1ac25b42769006740ba81a96b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:19.969177 13560 tablet_service.cc:1467] Processing CreateTablet for tablet a5969155f010461293a7f2f61c88c817 (DEFAULT_TABLE table=rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:19.973547 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5969155f010461293a7f2f61c88c817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:19.976177 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 257d72583cd94c0d8efdd8925be48dca (DEFAULT_TABLE table=rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:19.976997 13428 tablet_service.cc:1467] Processing CreateTablet for tablet a5969155f010461293a7f2f61c88c817 (DEFAULT_TABLE table=rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:19.977835 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5969155f010461293a7f2f61c88c817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:19.983307 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 257d72583cd94c0d8efdd8925be48dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:19.986482 14190 tablet_bootstrap.cc:492] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:19.986830 14191 tablet_bootstrap.cc:492] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:19.990125 14191 tablet_bootstrap.cc:654] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:19.990348 14190 tablet_bootstrap.cc:654] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:19.991271 14191 log.cc:824] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:19.997009 14190 log.cc:824] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:20.002424 14195 tablet_bootstrap.cc:492] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:20.004128 14191 tablet_bootstrap.cc:492] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:20.004417 14191 ts_tablet_manager.cc:1367] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.012s
I20240502 14:14:20.007354 14195 tablet_bootstrap.cc:654] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:20.008780 14195 log.cc:824] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:20.010416 14193 tablet_bootstrap.cc:492] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:20.011746 14190 tablet_bootstrap.cc:492] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:20.011749 14195 tablet_bootstrap.cc:492] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:20.012040 14190 ts_tablet_manager.cc:1367] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.007s
I20240502 14:14:20.012271 14195 ts_tablet_manager.cc:1367] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20240502 14:14:20.012432 14191 raft_consensus.cc:348] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.012871 14191 raft_consensus.cc:374] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:20.013005 14191 raft_consensus.cc:729] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:20.013581 14193 tablet_bootstrap.cc:654] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:20.013660 14191 consensus_queue.cc:260] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.014896 14193 log.cc:824] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:20.020581 14190 raft_consensus.cc:348] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.020529 14195 raft_consensus.cc:348] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.020934 14195 raft_consensus.cc:374] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:20.020963 14193 tablet_bootstrap.cc:492] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:20.020970 14190 raft_consensus.cc:374] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:20.021095 14190 raft_consensus.cc:729] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:20.021112 14195 raft_consensus.cc:729] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:20.021553 14193 ts_tablet_manager.cc:1367] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 14:14:20.021808 14195 consensus_queue.cc:260] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.022801 14190 consensus_queue.cc:260] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.027870 14191 ts_tablet_manager.cc:1398] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.023s user 0.018s sys 0.003s
I20240502 14:14:20.028276 14191 tablet_bootstrap.cc:492] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:20.029902 14191 tablet_bootstrap.cc:654] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:20.037827 14191 tablet_bootstrap.cc:492] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:20.038014 14191 ts_tablet_manager.cc:1367] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240502 14:14:20.038944 14191 raft_consensus.cc:348] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.039204 14191 raft_consensus.cc:374] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:20.039338 14191 raft_consensus.cc:729] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:20.039453 14193 raft_consensus.cc:348] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.039656 14191 consensus_queue.cc:260] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.040351 14191 ts_tablet_manager.cc:1398] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:20.040632 14191 tablet_bootstrap.cc:492] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:20.041293 14195 ts_tablet_manager.cc:1398] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.029s user 0.012s sys 0.012s
I20240502 14:14:20.041961 14195 tablet_bootstrap.cc:492] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:20.042127 14191 tablet_bootstrap.cc:654] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:20.045926 14190 ts_tablet_manager.cc:1398] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.034s user 0.025s sys 0.005s
I20240502 14:14:20.045961 14195 tablet_bootstrap.cc:654] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:20.046412 14190 tablet_bootstrap.cc:492] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:20.047076 14193 raft_consensus.cc:374] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:20.047261 14193 raft_consensus.cc:729] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:20.047922 14193 consensus_queue.cc:260] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.048483 14190 tablet_bootstrap.cc:654] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:20.051823 14193 ts_tablet_manager.cc:1398] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.030s user 0.008s sys 0.006s
I20240502 14:14:20.052379 14193 tablet_bootstrap.cc:492] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:20.054719 14193 tablet_bootstrap.cc:654] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:20.056735 14190 tablet_bootstrap.cc:492] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:20.056918 14190 ts_tablet_manager.cc:1367] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240502 14:14:20.057255 14191 tablet_bootstrap.cc:492] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:20.057426 14191 ts_tablet_manager.cc:1367] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.001s
I20240502 14:14:20.057544 14193 tablet_bootstrap.cc:492] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:20.057714 14193 ts_tablet_manager.cc:1367] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:14:20.058017 14195 tablet_bootstrap.cc:492] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:20.057924 14190 raft_consensus.cc:348] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:20.058157 14190 raft_consensus.cc:374] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:20.058178 14195 ts_tablet_manager.cc:1367] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.003s
I20240502 14:14:20.058133 14191 raft_consensus.cc:348] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:20.058311 14190 raft_consensus.cc:729] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:20.058382 14191 raft_consensus.cc:374] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:20.058494 14191 raft_consensus.cc:729] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:20.058607 14190 consensus_queue.cc:260] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:20.058607 14193 raft_consensus.cc:348] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.058784 14193 raft_consensus.cc:374] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:20.058787 14191 consensus_queue.cc:260] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:20.058991 14195 raft_consensus.cc:348] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:20.059124 14190 ts_tablet_manager.cc:1398] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:20.059211 14195 raft_consensus.cc:374] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:20.059412 14191 ts_tablet_manager.cc:1398] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:20.058913 14193 raft_consensus.cc:729] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:20.059314 14195 raft_consensus.cc:729] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:20.059651 14193 consensus_queue.cc:260] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.060256 14193 ts_tablet_manager.cc:1398] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:20.060784 14195 consensus_queue.cc:260] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:20.061412 14195 ts_tablet_manager.cc:1398] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:20.087723 14199 raft_consensus.cc:482] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:20.087910 14199 raft_consensus.cc:504] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.089304 14199 leader_election.cc:288] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:20.095703 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5969155f010461293a7f2f61c88c817" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:20.095870 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5969155f010461293a7f2f61c88c817" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:20.096141 13188 raft_consensus.cc:2454] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:20.096294 13449 raft_consensus.cc:2454] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:20.096695 13514 leader_election.cc:302] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:20.096983 14199 raft_consensus.cc:2781] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:20.097098 14199 raft_consensus.cc:482] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:20.097178 14199 raft_consensus.cc:3037] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:20.098515 14205 raft_consensus.cc:482] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:20.098626 14205 raft_consensus.cc:504] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.098994 14199 raft_consensus.cc:504] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.099503 14199 leader_election.cc:288] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:20.099905 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5969155f010461293a7f2f61c88c817" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:20.100073 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5969155f010461293a7f2f61c88c817" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:20.100106 13188 raft_consensus.cc:3037] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:20.100271 13449 raft_consensus.cc:3037] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:20.100278 14205 leader_election.cc:288] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:20.102913 13188 raft_consensus.cc:2454] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:20.103332 13514 leader_election.cc:302] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:20.103621 14199 raft_consensus.cc:2781] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:20.104423 13449 raft_consensus.cc:2454] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
W20240502 14:14:20.108309 13234 tablet.cc:2369] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:20.109408 14199 raft_consensus.cc:686] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:20.109874 14199 consensus_queue.cc:237] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.111662 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257d72583cd94c0d8efdd8925be48dca" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:20.112054 13319 raft_consensus.cc:2454] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:20.112746 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257d72583cd94c0d8efdd8925be48dca" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:20.112970 13188 raft_consensus.cc:2454] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:20.113010 13384 leader_election.cc:302] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:20.113431 14205 raft_consensus.cc:2781] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:20.113554 14205 raft_consensus.cc:482] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:20.113636 14205 raft_consensus.cc:3037] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:20.115909 14205 raft_consensus.cc:504] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
W20240502 14:14:20.116421 13627 tablet.cc:2369] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:20.116596 14205 leader_election.cc:288] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:20.116432 13044 catalog_manager.cc:5484] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:20.117345 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257d72583cd94c0d8efdd8925be48dca" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:20.117524 13319 raft_consensus.cc:3037] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:20.117538 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "257d72583cd94c0d8efdd8925be48dca" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:20.117722 13188 raft_consensus.cc:3037] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:20.120071 13188 raft_consensus.cc:2454] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:20.120114 13319 raft_consensus.cc:2454] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:20.120465 13382 leader_election.cc:302] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:20.120848 14205 raft_consensus.cc:2781] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:20.121500 14205 raft_consensus.cc:686] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:20.121958 14205 consensus_queue.cc:237] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:20.125705 13044 catalog_manager.cc:5484] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } }
W20240502 14:14:20.167851 13496 tablet.cc:2369] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:20.172417 14197 raft_consensus.cc:482] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:20.172591 14197 raft_consensus.cc:504] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:20.174091 14197 leader_election.cc:288] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:20.180402 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd031f1ac25b42769006740ba81a96b9" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:20.180616 13319 raft_consensus.cc:2454] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:20.180577 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd031f1ac25b42769006740ba81a96b9" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:20.180958 13581 raft_consensus.cc:2454] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:20.181026 13122 leader_election.cc:302] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:20.181419 14197 raft_consensus.cc:2781] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:20.181540 14197 raft_consensus.cc:482] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:20.181631 14197 raft_consensus.cc:3037] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:20.183527 14197 raft_consensus.cc:504] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:20.184036 14197 leader_election.cc:288] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:20.184361 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd031f1ac25b42769006740ba81a96b9" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:20.184509 13581 raft_consensus.cc:3037] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:20.184445 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd031f1ac25b42769006740ba81a96b9" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:20.184634 13319 raft_consensus.cc:3037] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:20.186404 13581 raft_consensus.cc:2454] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:20.186426 13319 raft_consensus.cc:2454] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:20.186836 13120 leader_election.cc:302] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:20.187137 14197 raft_consensus.cc:2781] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:20.187479 14197 raft_consensus.cc:686] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:20.187951 14197 consensus_queue.cc:237] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:20.191701 13044 catalog_manager.cc:5484] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:20.272536 13365 tablet.cc:2369] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:14:20.548148 14221 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:20.548401 14221 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:20.554790 14221 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 14:14:20.617300 14205 consensus_queue.cc:1035] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:20.625847 14205 consensus_queue.cc:1035] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:20.719693 14197 consensus_queue.cc:1035] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:20.736131 13449 raft_consensus.cc:1261] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:20.742650 14265 consensus_queue.cc:1035] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:20.743471 13188 raft_consensus.cc:1261] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:20.744614 14264 consensus_queue.cc:1035] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:20.783365 13319 raft_consensus.cc:1261] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:20.784152 14272 consensus_queue.cc:1035] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:20.802038 14257 mvcc.cc:205] Tried to move back new op lower bound from 7023244331793145856 to 7023244329416089600. Current Snapshot: MvccSnapshot[applied={T|T < 7023244331793145856}]
I20240502 14:14:20.824393 14249 mvcc.cc:205] Tried to move back new op lower bound from 7023244332072878080 to 7023244329732997120. Current Snapshot: MvccSnapshot[applied={T|T < 7023244332072878080}]
I20240502 14:14:20.838490 14268 mvcc.cc:205] Tried to move back new op lower bound from 7023244331793145856 to 7023244329416089600. Current Snapshot: MvccSnapshot[applied={T|T < 7023244331793145856}]
I20240502 14:14:20.861014 14248 mvcc.cc:205] Tried to move back new op lower bound from 7023244331793145856 to 7023244329416089600. Current Snapshot: MvccSnapshot[applied={T|T < 7023244331793145856}]
INSERT report
rows total: 2000
time total: 312.827 ms
time per row: 0.156413 ms
I20240502 14:14:21.001876 13043 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:21.009824 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 27857198d4ec430ebee897c90141c58d (DEFAULT_TABLE table=rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.009889 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 27857198d4ec430ebee897c90141c58d (DEFAULT_TABLE table=rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.010275 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27857198d4ec430ebee897c90141c58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.010180 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 27857198d4ec430ebee897c90141c58d (DEFAULT_TABLE table=rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.010303 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27857198d4ec430ebee897c90141c58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.010573 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27857198d4ec430ebee897c90141c58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.014529 14296 tablet_bootstrap.cc:492] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:21.015205 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6b71f7e77b4084a8904b1045099472 (DEFAULT_TABLE table=rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.015403 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 6395ce4c30eb4644b0a711de3e60b9c0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.015568 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6b71f7e77b4084a8904b1045099472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.015841 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6395ce4c30eb4644b0a711de3e60b9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.016183 14296 tablet_bootstrap.cc:654] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.016906 13167 tablet_service.cc:1467] Processing CreateTablet for tablet 6395ce4c30eb4644b0a711de3e60b9c0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.017328 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6395ce4c30eb4644b0a711de3e60b9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.020718 13428 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6b71f7e77b4084a8904b1045099472 (DEFAULT_TABLE table=rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.021104 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6b71f7e77b4084a8904b1045099472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.024242 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 8a6b71f7e77b4084a8904b1045099472 (DEFAULT_TABLE table=rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.024662 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6b71f7e77b4084a8904b1045099472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.024839 14300 tablet_bootstrap.cc:492] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:21.027792 14300 tablet_bootstrap.cc:654] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.028352 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 6395ce4c30eb4644b0a711de3e60b9c0 (DEFAULT_TABLE table=rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:21.028782 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6395ce4c30eb4644b0a711de3e60b9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:21.032110 14299 tablet_bootstrap.cc:492] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:21.034163 14299 tablet_bootstrap.cc:654] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.034894 14297 tablet_bootstrap.cc:492] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:21.036266 14296 tablet_bootstrap.cc:492] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:21.036454 14296 ts_tablet_manager.cc:1367] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.009s
I20240502 14:14:21.037416 14297 tablet_bootstrap.cc:654] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.037364 14296 raft_consensus.cc:348] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.037694 14299 tablet_bootstrap.cc:492] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:21.037721 14296 raft_consensus.cc:374] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.037807 14296 raft_consensus.cc:729] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.038130 14296 consensus_queue.cc:260] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.038391 14299 ts_tablet_manager.cc:1367] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:14:21.038826 14296 ts_tablet_manager.cc:1398] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:21.039140 14296 tablet_bootstrap.cc:492] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:21.039841 14299 raft_consensus.cc:348] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:21.040161 14299 raft_consensus.cc:374] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.040329 14299 raft_consensus.cc:729] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.040659 14296 tablet_bootstrap.cc:654] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.040710 14299 consensus_queue.cc:260] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:21.041620 14299 ts_tablet_manager.cc:1398] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:21.042984 14297 tablet_bootstrap.cc:492] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:21.043148 14297 ts_tablet_manager.cc:1367] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20240502 14:14:21.043256 14265 raft_consensus.cc:482] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:21.043392 14265 raft_consensus.cc:504] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.043816 14297 raft_consensus.cc:348] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.043943 14269 raft_consensus.cc:482] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:21.044006 14297 raft_consensus.cc:374] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.044081 14297 raft_consensus.cc:729] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.044052 14269 raft_consensus.cc:504] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:21.044137 14265 leader_election.cc:288] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:21.044764 14296 tablet_bootstrap.cc:492] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:21.044781 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27857198d4ec430ebee897c90141c58d" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:21.044828 14269 leader_election.cc:288] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:21.045015 14296 ts_tablet_manager.cc:1367] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:14:21.044332 14297 consensus_queue.cc:260] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.045334 13449 raft_consensus.cc:2454] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:21.045598 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6395ce4c30eb4644b0a711de3e60b9c0" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:21.046021 13514 leader_election.cc:302] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:21.046382 14265 raft_consensus.cc:2781] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:21.046484 14265 raft_consensus.cc:482] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240502 14:14:21.046416 13122 leader_election.cc:341] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:21.046591 14265 raft_consensus.cc:3037] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.046552 14296 raft_consensus.cc:348] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.046803 14296 raft_consensus.cc:374] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.046869 14296 raft_consensus.cc:729] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.047165 14296 consensus_queue.cc:260] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.048286 14296 ts_tablet_manager.cc:1398] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:21.049737 14265 raft_consensus.cc:504] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.051626 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27857198d4ec430ebee897c90141c58d" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:21.051838 13449 raft_consensus.cc:3037] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.054924 13449 raft_consensus.cc:2454] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:21.055425 13514 leader_election.cc:302] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:21.055773 14264 raft_consensus.cc:2781] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:21.055939 14264 raft_consensus.cc:686] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:21.056284 14264 consensus_queue.cc:237] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.058207 14265 leader_election.cc:288] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:21.058630 14297 ts_tablet_manager.cc:1398] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.015s user 0.000s sys 0.001s
I20240502 14:14:21.058873 14297 tablet_bootstrap.cc:492] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:21.058946 14300 tablet_bootstrap.cc:492] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:21.059059 14300 ts_tablet_manager.cc:1367] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.034s user 0.014s sys 0.000s
I20240502 14:14:21.060449 14297 tablet_bootstrap.cc:654] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.059796 14300 raft_consensus.cc:348] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:21.060962 14300 raft_consensus.cc:374] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.061100 14300 raft_consensus.cc:729] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.061517 14300 consensus_queue.cc:260] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:21.064677 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6395ce4c30eb4644b0a711de3e60b9c0" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:21.064550 13046 catalog_manager.cc:5484] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:21.065297 13120 leader_election.cc:341] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:21.065426 13120 leader_election.cc:302] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d
I20240502 14:14:21.065852 14269 raft_consensus.cc:2726] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:21.066551 14300 ts_tablet_manager.cc:1398] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:14:21.068176 14300 tablet_bootstrap.cc:492] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:21.068421 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27857198d4ec430ebee897c90141c58d" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:21.068665 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27857198d4ec430ebee897c90141c58d" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
W20240502 14:14:21.069116 13516 leader_election.cc:341] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:14:21.069507 13516 leader_election.cc:341] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:21.069686 14300 tablet_bootstrap.cc:654] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.071204 14300 tablet_bootstrap.cc:492] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:21.071275 14297 tablet_bootstrap.cc:492] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:21.071357 14300 ts_tablet_manager.cc:1367] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:21.071424 14297 ts_tablet_manager.cc:1367] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20240502 14:14:21.072058 14297 raft_consensus.cc:348] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.072100 14300 raft_consensus.cc:348] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.072199 14297 raft_consensus.cc:374] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.072261 14297 raft_consensus.cc:729] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.072309 14300 raft_consensus.cc:374] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.072392 14300 raft_consensus.cc:729] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.072485 14297 consensus_queue.cc:260] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.072636 14300 consensus_queue.cc:260] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.072911 14297 ts_tablet_manager.cc:1398] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:14:21.073154 14300 ts_tablet_manager.cc:1398] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:21.073518 14300 tablet_bootstrap.cc:492] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:21.073272 14297 tablet_bootstrap.cc:492] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:21.074770 14300 tablet_bootstrap.cc:654] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.075207 14297 tablet_bootstrap.cc:654] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:21.076615 14300 tablet_bootstrap.cc:492] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:21.076776 14300 ts_tablet_manager.cc:1367] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:21.077306 14297 tablet_bootstrap.cc:492] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:21.077425 14297 ts_tablet_manager.cc:1367] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 14:14:21.077679 14300 raft_consensus.cc:348] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.077839 14300 raft_consensus.cc:374] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.077925 14300 raft_consensus.cc:729] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.078055 14297 raft_consensus.cc:348] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:21.078217 14297 raft_consensus.cc:374] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:21.078300 14297 raft_consensus.cc:729] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:21.078198 14300 consensus_queue.cc:260] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.078490 14297 consensus_queue.cc:260] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:21.078737 14300 ts_tablet_manager.cc:1398] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:21.078943 14297 ts_tablet_manager.cc:1398] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:14:21.171633 14245 raft_consensus.cc:482] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:21.171842 14245 raft_consensus.cc:504] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:21.173406 14245 leader_election.cc:288] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:21.179543 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6395ce4c30eb4644b0a711de3e60b9c0" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:21.179791 13188 raft_consensus.cc:2454] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:21.179792 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6395ce4c30eb4644b0a711de3e60b9c0" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:21.180044 13449 raft_consensus.cc:2454] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:21.180351 13251 leader_election.cc:302] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:21.180673 14245 raft_consensus.cc:2781] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:21.180789 14245 raft_consensus.cc:482] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:21.180882 14245 raft_consensus.cc:3037] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.182854 14245 raft_consensus.cc:504] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:21.183434 14245 leader_election.cc:288] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:21.183791 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6395ce4c30eb4644b0a711de3e60b9c0" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:21.183902 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6395ce4c30eb4644b0a711de3e60b9c0" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:21.184000 13188 raft_consensus.cc:3037] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.184094 13449 raft_consensus.cc:3037] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.186858 13188 raft_consensus.cc:2454] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:21.186858 13449 raft_consensus.cc:2454] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:21.187224 13251 leader_election.cc:302] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:21.187464 14245 raft_consensus.cc:2781] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:21.187806 14245 raft_consensus.cc:686] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:21.188208 14245 consensus_queue.cc:237] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:21.188704 14315 raft_consensus.cc:482] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:21.188838 14315 raft_consensus.cc:504] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.189602 14315 leader_election.cc:288] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:21.189865 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6b71f7e77b4084a8904b1045099472" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:21.190124 13449 raft_consensus.cc:2454] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:21.190971 13251 leader_election.cc:302] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:21.191201 14315 raft_consensus.cc:2781] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:21.191301 14315 raft_consensus.cc:482] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:21.191385 14315 raft_consensus.cc:3037] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.192821 13046 catalog_manager.cc:5484] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:21.193645 14315 raft_consensus.cc:504] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.194185 14315 leader_election.cc:288] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:21.195569 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6b71f7e77b4084a8904b1045099472" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:21.195689 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6b71f7e77b4084a8904b1045099472" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:21.195875 13581 raft_consensus.cc:2454] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:21.195436 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a6b71f7e77b4084a8904b1045099472" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:21.196273 13449 raft_consensus.cc:3037] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.198304 13449 raft_consensus.cc:2454] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:21.198609 13251 leader_election.cc:302] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters: 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:14:21.198910 14315 raft_consensus.cc:2781] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:21.199026 14315 raft_consensus.cc:686] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:21.199255 14315 consensus_queue.cc:237] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:21.201431 13046 catalog_manager.cc:5484] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:21.502082 13318 raft_consensus.cc:3037] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.504398 14264 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:21.509867 14260 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:21.523356 14318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:21.523587 14318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:21.530577 14318 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 14:14:21.595625 14315 consensus_queue.cc:1035] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:21.602337 14245 consensus_queue.cc:1035] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:21.620920 13580 raft_consensus.cc:3037] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:21.624121 14245 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:21.642189 14315 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 229.242 ms
time per row: 0.114621 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:14:22.741480 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:22.749392 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9a5846fbad47c39a8a197e26b48080 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.749395 13167 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9a5846fbad47c39a8a197e26b48080 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.749754 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9a5846fbad47c39a8a197e26b48080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.749810 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9a5846fbad47c39a8a197e26b48080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.750396 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 2f9a5846fbad47c39a8a197e26b48080 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.750501 13298 tablet_service.cc:1467] Processing CreateTablet for tablet e293ac3e623c432e8ce57333b74a6386 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.750778 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f9a5846fbad47c39a8a197e26b48080. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.750954 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e293ac3e623c432e8ce57333b74a6386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.757802 14383 tablet_bootstrap.cc:492] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:22.757967 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 0395e4ffbe20447d8dbc0722f4c74994 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.758452 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0395e4ffbe20447d8dbc0722f4c74994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.759685 14383 tablet_bootstrap.cc:654] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.762054 13561 tablet_service.cc:1467] Processing CreateTablet for tablet 0395e4ffbe20447d8dbc0722f4c74994 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.762171 13559 tablet_service.cc:1467] Processing CreateTablet for tablet e293ac3e623c432e8ce57333b74a6386 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.762532 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0395e4ffbe20447d8dbc0722f4c74994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.763041 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 0395e4ffbe20447d8dbc0722f4c74994 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.763366 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0395e4ffbe20447d8dbc0722f4c74994. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.763861 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e293ac3e623c432e8ce57333b74a6386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.765751 13166 tablet_service.cc:1467] Processing CreateTablet for tablet e293ac3e623c432e8ce57333b74a6386 (DEFAULT_TABLE table=rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:22.766227 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e293ac3e623c432e8ce57333b74a6386. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:22.781354 14381 tablet_bootstrap.cc:492] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:22.783577 14381 tablet_bootstrap.cc:654] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.784628 14385 tablet_bootstrap.cc:492] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:22.785705 14383 tablet_bootstrap.cc:492] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:22.786000 14383 ts_tablet_manager.cc:1367] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.028s user 0.018s sys 0.009s
I20240502 14:14:22.786636 14385 tablet_bootstrap.cc:654] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.787077 14383 raft_consensus.cc:348] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:22.787412 14383 raft_consensus.cc:374] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.787554 14383 raft_consensus.cc:729] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.787855 14383 consensus_queue.cc:260] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:22.788566 14383 ts_tablet_manager.cc:1398] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:22.788823 14383 tablet_bootstrap.cc:492] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:22.789181 14382 tablet_bootstrap.cc:492] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:22.789361 14381 tablet_bootstrap.cc:492] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:22.789499 14381 ts_tablet_manager.cc:1367] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 14:14:22.790195 14381 raft_consensus.cc:348] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:22.790383 14381 raft_consensus.cc:374] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.790484 14381 raft_consensus.cc:729] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.790700 14381 consensus_queue.cc:260] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:22.790984 14382 tablet_bootstrap.cc:654] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.791901 14381 ts_tablet_manager.cc:1398] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:22.792177 14381 tablet_bootstrap.cc:492] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:22.793273 14385 tablet_bootstrap.cc:492] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:22.793426 14385 ts_tablet_manager.cc:1367] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240502 14:14:22.794275 14385 raft_consensus.cc:348] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:22.794505 14385 raft_consensus.cc:374] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.794603 14385 raft_consensus.cc:729] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.794710 14381 tablet_bootstrap.cc:654] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.794792 14385 consensus_queue.cc:260] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:22.795272 14385 ts_tablet_manager.cc:1398] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:22.797390 14383 tablet_bootstrap.cc:654] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.800619 14383 tablet_bootstrap.cc:492] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:22.800721 14381 tablet_bootstrap.cc:492] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:22.800760 14383 ts_tablet_manager.cc:1367] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.004s
I20240502 14:14:22.800874 14381 ts_tablet_manager.cc:1367] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240502 14:14:22.801311 14383 raft_consensus.cc:348] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:22.801465 14383 raft_consensus.cc:374] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.801534 14383 raft_consensus.cc:729] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.801554 14382 tablet_bootstrap.cc:492] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:22.801510 14381 raft_consensus.cc:348] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:22.801663 14382 ts_tablet_manager.cc:1367] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20240502 14:14:22.801731 14381 raft_consensus.cc:374] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.801810 14381 raft_consensus.cc:729] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.801738 14383 consensus_queue.cc:260] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:22.802032 14381 consensus_queue.cc:260] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:22.802155 14383 ts_tablet_manager.cc:1398] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:14:22.802553 14381 ts_tablet_manager.cc:1398] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:22.802829 14381 tablet_bootstrap.cc:492] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:22.803011 14382 raft_consensus.cc:348] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:22.803236 14382 raft_consensus.cc:374] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.803351 14382 raft_consensus.cc:729] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.803618 14382 consensus_queue.cc:260] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:22.804114 14381 tablet_bootstrap.cc:654] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.805161 14382 ts_tablet_manager.cc:1398] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:14:22.805799 14382 tablet_bootstrap.cc:492] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:22.809336 14382 tablet_bootstrap.cc:654] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.810221 14381 tablet_bootstrap.cc:492] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:22.810356 14381 ts_tablet_manager.cc:1367] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 14:14:22.810856 14382 tablet_bootstrap.cc:492] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:22.810842 14381 raft_consensus.cc:348] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:22.810978 14382 ts_tablet_manager.cc:1367] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 14:14:22.811013 14381 raft_consensus.cc:374] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.811107 14381 raft_consensus.cc:729] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.811300 14381 consensus_queue.cc:260] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:22.811502 14382 raft_consensus.cc:348] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:22.811691 14381 ts_tablet_manager.cc:1398] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:14:22.811708 14382 raft_consensus.cc:374] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.811803 14382 raft_consensus.cc:729] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.812002 14382 consensus_queue.cc:260] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:22.812459 14382 ts_tablet_manager.cc:1398] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:14:22.812723 14382 tablet_bootstrap.cc:492] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:22.814020 14382 tablet_bootstrap.cc:654] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:22.815785 14382 tablet_bootstrap.cc:492] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:22.815919 14382 ts_tablet_manager.cc:1367] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 14:14:22.816408 14382 raft_consensus.cc:348] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:22.816545 14382 raft_consensus.cc:374] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:22.816637 14382 raft_consensus.cc:729] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:22.816823 14382 consensus_queue.cc:260] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:22.817246 14382 ts_tablet_manager.cc:1398] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:14:22.875572 14269 raft_consensus.cc:482] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:22.875713 14269 raft_consensus.cc:504] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:22.876576 14269 leader_election.cc:288] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:22.877024 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9a5846fbad47c39a8a197e26b48080" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:22.877197 13580 raft_consensus.cc:2454] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:22.877127 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9a5846fbad47c39a8a197e26b48080" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:22.877400 13318 raft_consensus.cc:2454] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:22.877624 13120 leader_election.cc:302] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:22.877893 14269 raft_consensus.cc:2781] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:22.877990 14269 raft_consensus.cc:482] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:22.878082 14269 raft_consensus.cc:3037] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.879724 14269 raft_consensus.cc:504] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:22.880266 14269 leader_election.cc:288] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:22.880666 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9a5846fbad47c39a8a197e26b48080" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:22.880779 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f9a5846fbad47c39a8a197e26b48080" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:22.880864 13580 raft_consensus.cc:3037] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.880987 13318 raft_consensus.cc:3037] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.883481 13580 raft_consensus.cc:2454] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:22.883556 13318 raft_consensus.cc:2454] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:22.883877 13120 leader_election.cc:302] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:22.884164 14269 raft_consensus.cc:2781] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:22.884501 14269 raft_consensus.cc:686] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:22.884765 14269 consensus_queue.cc:237] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:22.887759 13045 catalog_manager.cc:5484] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:22.935462 14357 raft_consensus.cc:482] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:22.935595 14357 raft_consensus.cc:504] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:22.936167 14357 leader_election.cc:288] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:22.936589 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e293ac3e623c432e8ce57333b74a6386" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:22.936648 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e293ac3e623c432e8ce57333b74a6386" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:22.936771 13188 raft_consensus.cc:2454] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:22.936882 13580 raft_consensus.cc:2454] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:22.937213 13251 leader_election.cc:302] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:22.937495 14357 raft_consensus.cc:2781] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:22.937618 14357 raft_consensus.cc:482] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:22.937693 14357 raft_consensus.cc:3037] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.939282 14357 raft_consensus.cc:504] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:22.939683 14357 leader_election.cc:288] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:22.940075 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e293ac3e623c432e8ce57333b74a6386" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:22.940131 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e293ac3e623c432e8ce57333b74a6386" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:22.940213 13188 raft_consensus.cc:3037] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.940286 13580 raft_consensus.cc:3037] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.941874 13580 raft_consensus.cc:2454] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:22.941900 13188 raft_consensus.cc:2454] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:22.942175 13251 leader_election.cc:302] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:22.942396 14357 raft_consensus.cc:2781] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:22.942675 14205 raft_consensus.cc:482] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:22.942724 14357 raft_consensus.cc:686] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:22.942858 14205 raft_consensus.cc:504] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:22.942968 14357 consensus_queue.cc:237] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:22.943548 14205 leader_election.cc:288] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:22.945428 13045 catalog_manager.cc:5484] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:22.946017 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0395e4ffbe20447d8dbc0722f4c74994" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:22.946204 13188 raft_consensus.cc:2454] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:22.946558 13382 leader_election.cc:302] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:22.946817 14205 raft_consensus.cc:2781] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:22.946904 14205 raft_consensus.cc:482] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:22.947032 14205 raft_consensus.cc:3037] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.949128 14205 raft_consensus.cc:504] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:22.949653 14205 leader_election.cc:288] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:22.950037 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0395e4ffbe20447d8dbc0722f4c74994" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:22.950183 13188 raft_consensus.cc:3037] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:22.951642 13188 raft_consensus.cc:2454] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:22.951985 13382 leader_election.cc:302] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:22.952278 14205 raft_consensus.cc:2781] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:22.953431 14205 raft_consensus.cc:686] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:22.953765 14205 consensus_queue.cc:237] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:22.956282 13045 catalog_manager.cc:5484] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:22.961937 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0395e4ffbe20447d8dbc0722f4c74994" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:22.961951 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0395e4ffbe20447d8dbc0722f4c74994" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:22.962136 13580 raft_consensus.cc:2454] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
W20240502 14:14:23.304355 14402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:23.304663 14402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:14:23.310336 14269 consensus_queue.cc:1035] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:23.312604 14402 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:23.314927 14394 consensus_queue.cc:1035] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:23.326858 14357 consensus_queue.cc:1035] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:23.331214 14396 consensus_queue.cc:1035] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_2'
I20240502 14:14:23.451498 14205 consensus_queue.cc:1035] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:23.488664 13580 raft_consensus.cc:3037] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:23.491367 13580 raft_consensus.cc:1261] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:23.493935 14205 consensus_queue.cc:1035] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 228.212 ms
time per row: 0.114106 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:14:24.570101 13044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:24.578564 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5329f38f3343baaf43265990d86938 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8fa288258a0f4231be6de034088616ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.579017 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5329f38f3343baaf43265990d86938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.579221 13167 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5329f38f3343baaf43265990d86938 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8fa288258a0f4231be6de034088616ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.579238 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a (DEFAULT_TABLE table=rebalance_test_table_3 [id=8fa288258a0f4231be6de034088616ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.579629 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5329f38f3343baaf43265990d86938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.579633 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.580945 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a (DEFAULT_TABLE table=rebalance_test_table_3 [id=8fa288258a0f4231be6de034088616ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.581857 13428 tablet_service.cc:1467] Processing CreateTablet for tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a (DEFAULT_TABLE table=rebalance_test_table_3 [id=8fa288258a0f4231be6de034088616ab]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.582216 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.582226 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2fc48c889b481e9b4d249819a196a0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8fa288258a0f4231be6de034088616ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.582939 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2fc48c889b481e9b4d249819a196a0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8fa288258a0f4231be6de034088616ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.583089 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 0a2fc48c889b481e9b4d249819a196a0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8fa288258a0f4231be6de034088616ab]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.583395 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2fc48c889b481e9b4d249819a196a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.583545 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 9c5329f38f3343baaf43265990d86938 (DEFAULT_TABLE table=rebalance_test_table_3 [id=8fa288258a0f4231be6de034088616ab]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:24.583920 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5329f38f3343baaf43265990d86938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.589468 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.589893 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2fc48c889b481e9b4d249819a196a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.596539 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2fc48c889b481e9b4d249819a196a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:24.597222 14476 tablet_bootstrap.cc:492] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:24.599542 14476 tablet_bootstrap.cc:654] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:24.610023 14478 tablet_bootstrap.cc:492] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:24.612089 14478 tablet_bootstrap.cc:654] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:24.615332 14480 tablet_bootstrap.cc:492] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:24.619334 14480 tablet_bootstrap.cc:654] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:24.620910 14478 tablet_bootstrap.cc:492] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:24.621172 14478 ts_tablet_manager.cc:1367] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20240502 14:14:24.622072 14478 raft_consensus.cc:348] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.622426 14478 raft_consensus.cc:374] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:24.622576 14478 raft_consensus.cc:729] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:24.622934 14478 consensus_queue.cc:260] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.623600 14478 ts_tablet_manager.cc:1398] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:24.623981 14264 raft_consensus.cc:482] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:24.624115 14264 raft_consensus.cc:504] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.624644 14264 leader_election.cc:288] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:24.625126 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2fc48c889b481e9b4d249819a196a0" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:24.625294 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2fc48c889b481e9b4d249819a196a0" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
W20240502 14:14:24.625829 13514 leader_election.cc:341] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:14:24.626395 13514 leader_election.cc:341] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:24.626586 13514 leader_election.cc:302] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:14:24.626900 14264 raft_consensus.cc:2726] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:24.629710 14480 tablet_bootstrap.cc:492] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:24.629714 14478 tablet_bootstrap.cc:492] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:24.630198 14480 ts_tablet_manager.cc:1367] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240502 14:14:24.630862 14475 tablet_bootstrap.cc:492] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:24.631139 14478 tablet_bootstrap.cc:654] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:24.631100 14480 raft_consensus.cc:348] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.631270 14480 raft_consensus.cc:374] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:24.631359 14480 raft_consensus.cc:729] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:24.632442 14478 tablet_bootstrap.cc:492] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:24.632566 14478 ts_tablet_manager.cc:1367] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:24.632573 14475 tablet_bootstrap.cc:654] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:24.632647 14480 consensus_queue.cc:260] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.633049 14478 raft_consensus.cc:348] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:24.633208 14478 raft_consensus.cc:374] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:24.633337 14478 raft_consensus.cc:729] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:24.633643 14480 ts_tablet_manager.cc:1398] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:24.633621 14478 consensus_queue.cc:260] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:24.634107 14478 ts_tablet_manager.cc:1398] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 14:14:24.634801 14476 tablet_bootstrap.cc:492] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:24.635004 14476 ts_tablet_manager.cc:1367] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.020s
I20240502 14:14:24.635911 14476 raft_consensus.cc:348] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:24.636180 14476 raft_consensus.cc:374] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:24.636276 14476 raft_consensus.cc:729] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:24.636583 14476 consensus_queue.cc:260] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:24.637373 14476 ts_tablet_manager.cc:1398] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:24.637614 14476 tablet_bootstrap.cc:492] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:24.637897 14480 tablet_bootstrap.cc:492] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:24.639273 14476 tablet_bootstrap.cc:654] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:24.640388 14475 tablet_bootstrap.cc:492] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:24.640516 14475 ts_tablet_manager.cc:1367] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240502 14:14:24.641108 14476 tablet_bootstrap.cc:492] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:24.641263 14476 ts_tablet_manager.cc:1367] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:24.641407 14480 tablet_bootstrap.cc:654] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:24.641419 14475 raft_consensus.cc:348] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:24.641592 14475 raft_consensus.cc:374] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:24.641681 14475 raft_consensus.cc:729] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:24.641871 14476 raft_consensus.cc:348] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.641872 14475 consensus_queue.cc:260] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:24.642047 14476 raft_consensus.cc:374] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:24.642122 14476 raft_consensus.cc:729] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:24.642325 14476 consensus_queue.cc:260] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.642807 14476 ts_tablet_manager.cc:1398] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:14:24.643090 14476 tablet_bootstrap.cc:492] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:24.643316 14475 ts_tablet_manager.cc:1398] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:24.643599 14475 tablet_bootstrap.cc:492] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:24.644492 14476 tablet_bootstrap.cc:654] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:24.645937 14475 tablet_bootstrap.cc:654] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:24.649506 14480 tablet_bootstrap.cc:492] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:24.649689 14480 ts_tablet_manager.cc:1367] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240502 14:14:24.650208 14475 tablet_bootstrap.cc:492] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:24.650314 14476 tablet_bootstrap.cc:492] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:24.650332 14475 ts_tablet_manager.cc:1367] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240502 14:14:24.650439 14476 ts_tablet_manager.cc:1367] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240502 14:14:24.650350 14480 raft_consensus.cc:348] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.650525 14480 raft_consensus.cc:374] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:24.650604 14480 raft_consensus.cc:729] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:24.650810 14480 consensus_queue.cc:260] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.650884 14475 raft_consensus.cc:348] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.651053 14475 raft_consensus.cc:374] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:24.651140 14475 raft_consensus.cc:729] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:24.651127 14476 raft_consensus.cc:348] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.651270 14480 ts_tablet_manager.cc:1398] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:14:24.651352 14476 raft_consensus.cc:374] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:24.651446 14476 raft_consensus.cc:729] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:24.651351 14475 consensus_queue.cc:260] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.651777 14475 ts_tablet_manager.cc:1398] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:14:24.652117 14476 consensus_queue.cc:260] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.652612 14476 ts_tablet_manager.cc:1398] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:24.702994 14461 raft_consensus.cc:482] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:24.703117 14461 raft_consensus.cc:504] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.703681 14461 leader_election.cc:288] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:24.704000 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5329f38f3343baaf43265990d86938" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:24.704108 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5329f38f3343baaf43265990d86938" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:24.704228 13318 raft_consensus.cc:2454] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:24.704282 13186 raft_consensus.cc:2454] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:24.704591 13382 leader_election.cc:302] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:24.704804 14461 raft_consensus.cc:2781] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:24.704905 14461 raft_consensus.cc:482] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:24.704986 14461 raft_consensus.cc:3037] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:24.706681 14461 raft_consensus.cc:504] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.707216 14461 leader_election.cc:288] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:24.707588 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5329f38f3343baaf43265990d86938" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:24.707695 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c5329f38f3343baaf43265990d86938" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:24.707782 13318 raft_consensus.cc:3037] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:24.707902 13186 raft_consensus.cc:3037] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:24.709455 13318 raft_consensus.cc:2454] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:24.709779 13384 leader_election.cc:302] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:24.709805 13186 raft_consensus.cc:2454] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:24.710012 14461 raft_consensus.cc:2781] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:24.710386 14461 raft_consensus.cc:686] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:24.710656 14461 consensus_queue.cc:237] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.712604 13044 catalog_manager.cc:5484] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:24.744485 14269 raft_consensus.cc:482] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:24.744593 14269 raft_consensus.cc:504] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.745013 14269 leader_election.cc:288] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:24.745332 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2fc48c889b481e9b4d249819a196a0" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:24.745529 13448 raft_consensus.cc:2454] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:24.745499 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2fc48c889b481e9b4d249819a196a0" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:24.745707 13580 raft_consensus.cc:2454] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:24.745829 13120 leader_election.cc:302] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:24.746102 14269 raft_consensus.cc:2781] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:24.746201 14269 raft_consensus.cc:482] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:24.746290 14269 raft_consensus.cc:3037] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:24.747941 14269 raft_consensus.cc:504] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.748358 14269 leader_election.cc:288] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:24.748708 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2fc48c889b481e9b4d249819a196a0" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:24.748768 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a2fc48c889b481e9b4d249819a196a0" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:24.748934 13580 raft_consensus.cc:3037] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:24.748937 13448 raft_consensus.cc:3037] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:24.751339 13448 raft_consensus.cc:2454] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:24.751339 13580 raft_consensus.cc:2454] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:24.751682 13120 leader_election.cc:302] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:24.751914 14269 raft_consensus.cc:2781] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:24.752183 14269 raft_consensus.cc:686] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:24.752408 14269 consensus_queue.cc:237] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:24.754421 13044 catalog_manager.cc:5484] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:24.865597 14357 raft_consensus.cc:482] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:24.865733 14357 raft_consensus.cc:504] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:24.866251 14357 leader_election.cc:288] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:24.866667 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ac2a9fa2154a0bbbe15aeaf178ea1a" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:24.866745 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ac2a9fa2154a0bbbe15aeaf178ea1a" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:24.866856 13448 raft_consensus.cc:2454] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:24.866991 13580 raft_consensus.cc:2454] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:24.867261 13251 leader_election.cc:302] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:24.867501 14357 raft_consensus.cc:2781] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:24.867591 14357 raft_consensus.cc:482] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:24.867681 14357 raft_consensus.cc:3037] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:24.869402 14357 raft_consensus.cc:504] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:24.869846 14357 leader_election.cc:288] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:24.870218 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ac2a9fa2154a0bbbe15aeaf178ea1a" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:24.870258 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21ac2a9fa2154a0bbbe15aeaf178ea1a" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:24.870363 13448 raft_consensus.cc:3037] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:24.870391 13580 raft_consensus.cc:3037] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:24.871883 13448 raft_consensus.cc:2454] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:24.871956 13580 raft_consensus.cc:2454] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:24.872212 13251 leader_election.cc:302] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:24.872663 14357 raft_consensus.cc:2781] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:24.872975 14357 raft_consensus.cc:686] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:24.873245 14357 consensus_queue.cc:237] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:24.875353 13044 catalog_manager.cc:5484] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:25.158402 14269 consensus_queue.cc:1035] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:25.166379 14269 consensus_queue.cc:1035] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:25.189718 14494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:25.190001 14494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:25.196252 14494 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 14:14:25.224215 14461 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:25.229002 14518 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:25.351218 13580 raft_consensus.cc:1261] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:25.351775 14492 consensus_queue.cc:1035] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:25.352748 13448 raft_consensus.cc:1261] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:25.353749 14492 consensus_queue.cc:1035] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 262.934 ms
time per row: 0.131467 ms
I20240502 14:14:25.592455 13045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:25.600298 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc40c95ebb94109a784c659b5b97ffe (DEFAULT_TABLE table=rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:25.600764 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc40c95ebb94109a784c659b5b97ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:25.600708 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc40c95ebb94109a784c659b5b97ffe (DEFAULT_TABLE table=rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:25.601112 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc40c95ebb94109a784c659b5b97ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:25.601635 13167 tablet_service.cc:1467] Processing CreateTablet for tablet 7cc40c95ebb94109a784c659b5b97ffe (DEFAULT_TABLE table=rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:25.601696 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2f3557f96d4f33b02941d086de1391 (DEFAULT_TABLE table=rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:25.601773 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2f3557f96d4f33b02941d086de1391 (DEFAULT_TABLE table=rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:25.602003 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7cc40c95ebb94109a784c659b5b97ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:25.602087 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2f3557f96d4f33b02941d086de1391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:25.602084 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2f3557f96d4f33b02941d086de1391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:25.605643 13560 tablet_service.cc:1467] Processing CreateTablet for tablet ad1c998f38c049eda8d4ff28fb2eca29 (DEFAULT_TABLE table=rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:25.606014 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1c998f38c049eda8d4ff28fb2eca29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:25.606611 13166 tablet_service.cc:1467] Processing CreateTablet for tablet ad1c998f38c049eda8d4ff28fb2eca29 (DEFAULT_TABLE table=rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:25.606984 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1c998f38c049eda8d4ff28fb2eca29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:25.610309 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 9b2f3557f96d4f33b02941d086de1391 (DEFAULT_TABLE table=rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:25.610368 13297 tablet_service.cc:1467] Processing CreateTablet for tablet ad1c998f38c049eda8d4ff28fb2eca29 (DEFAULT_TABLE table=rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:25.610670 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b2f3557f96d4f33b02941d086de1391. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:25.610849 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad1c998f38c049eda8d4ff28fb2eca29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:25.611871 14568 tablet_bootstrap.cc:492] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:25.613817 14568 tablet_bootstrap.cc:654] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.621592 14570 tablet_bootstrap.cc:492] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:25.622988 14568 tablet_bootstrap.cc:492] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:25.623122 14568 ts_tablet_manager.cc:1367] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240502 14:14:25.623502 14570 tablet_bootstrap.cc:654] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.623917 14568 raft_consensus.cc:348] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:25.624135 14568 raft_consensus.cc:374] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.624243 14568 raft_consensus.cc:729] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.624482 14568 consensus_queue.cc:260] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:25.625160 14568 ts_tablet_manager.cc:1398] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:25.625957 14568 tablet_bootstrap.cc:492] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:25.628016 14568 tablet_bootstrap.cc:654] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.632617 14570 tablet_bootstrap.cc:492] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:25.632862 14570 ts_tablet_manager.cc:1367] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20240502 14:14:25.633893 14570 raft_consensus.cc:348] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:25.634248 14570 raft_consensus.cc:374] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.634411 14570 raft_consensus.cc:729] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.634796 14570 consensus_queue.cc:260] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:25.638195 14570 ts_tablet_manager.cc:1398] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:25.642228 14570 tablet_bootstrap.cc:492] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:25.644254 14570 tablet_bootstrap.cc:654] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.647330 14573 tablet_bootstrap.cc:492] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:25.647805 14570 tablet_bootstrap.cc:492] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:25.647881 14567 tablet_bootstrap.cc:492] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:25.647953 14570 ts_tablet_manager.cc:1367] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:14:25.648551 14568 tablet_bootstrap.cc:492] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:25.648764 14568 ts_tablet_manager.cc:1367] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.018s
I20240502 14:14:25.648721 14570 raft_consensus.cc:348] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.648947 14570 raft_consensus.cc:374] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.649075 14570 raft_consensus.cc:729] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.649384 14570 consensus_queue.cc:260] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.649602 14567 tablet_bootstrap.cc:654] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.649536 14568 raft_consensus.cc:348] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.649783 14568 raft_consensus.cc:374] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.649886 14568 raft_consensus.cc:729] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.650267 14573 tablet_bootstrap.cc:654] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.650283 14570 ts_tablet_manager.cc:1398] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:25.650187 14568 consensus_queue.cc:260] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.650862 14568 ts_tablet_manager.cc:1398] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:25.651270 14570 tablet_bootstrap.cc:492] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:25.652773 14570 tablet_bootstrap.cc:654] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.655022 14567 tablet_bootstrap.cc:492] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:25.655171 14567 ts_tablet_manager.cc:1367] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240502 14:14:25.656123 14518 raft_consensus.cc:482] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:25.656076 14567 raft_consensus.cc:348] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:25.656345 14567 raft_consensus.cc:374] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.656319 14518 raft_consensus.cc:504] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.656448 14567 raft_consensus.cc:729] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.656728 14567 consensus_queue.cc:260] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:25.657218 14518 leader_election.cc:288] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:25.657382 14567 ts_tablet_manager.cc:1398] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:25.657966 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc40c95ebb94109a784c659b5b97ffe" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:25.658241 14567 tablet_bootstrap.cc:492] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:25.658195 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc40c95ebb94109a784c659b5b97ffe" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
W20240502 14:14:25.658674 13384 leader_election.cc:341] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:14:25.658910 13382 leader_election.cc:341] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:25.659009 13382 leader_election.cc:302] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d
I20240502 14:14:25.659417 14518 raft_consensus.cc:2726] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:25.659576 14573 tablet_bootstrap.cc:492] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:25.659760 14573 ts_tablet_manager.cc:1367] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.003s
I20240502 14:14:25.660671 14573 raft_consensus.cc:348] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.660964 14573 raft_consensus.cc:374] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.661120 14570 tablet_bootstrap.cc:492] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:25.661207 14573 raft_consensus.cc:729] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.661324 14570 ts_tablet_manager.cc:1367] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 14:14:25.661545 14573 consensus_queue.cc:260] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.662186 14570 raft_consensus.cc:348] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.662422 14570 raft_consensus.cc:374] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.662549 14570 raft_consensus.cc:729] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.662811 14570 consensus_queue.cc:260] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.663341 14570 ts_tablet_manager.cc:1398] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:25.664286 14573 ts_tablet_manager.cc:1398] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:25.665055 14573 tablet_bootstrap.cc:492] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:25.666584 14573 tablet_bootstrap.cc:654] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.668331 14567 tablet_bootstrap.cc:654] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:25.669171 14541 raft_consensus.cc:482] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:25.669324 14541 raft_consensus.cc:504] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:25.669610 14573 tablet_bootstrap.cc:492] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:25.669728 14573 ts_tablet_manager.cc:1367] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:14:25.670001 14541 leader_election.cc:288] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:25.670500 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2f3557f96d4f33b02941d086de1391" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:25.670606 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2f3557f96d4f33b02941d086de1391" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:25.670677 13448 raft_consensus.cc:2454] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:25.670680 14573 raft_consensus.cc:348] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.670806 13319 raft_consensus.cc:2454] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:25.670874 14573 raft_consensus.cc:374] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.671000 14573 raft_consensus.cc:729] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.671023 13514 leader_election.cc:302] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:25.671240 14541 raft_consensus.cc:2781] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:25.671355 14541 raft_consensus.cc:482] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:25.671298 14573 consensus_queue.cc:260] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.671433 14541 raft_consensus.cc:3037] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.672194 14573 ts_tablet_manager.cc:1398] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:25.673045 14567 tablet_bootstrap.cc:492] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:25.673169 14567 ts_tablet_manager.cc:1367] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20240502 14:14:25.673584 14541 raft_consensus.cc:504] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:25.674336 14567 raft_consensus.cc:348] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.674523 14567 raft_consensus.cc:374] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:25.674516 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2f3557f96d4f33b02941d086de1391" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:25.674592 14567 raft_consensus.cc:729] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:25.674695 13448 raft_consensus.cc:3037] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.674854 14567 consensus_queue.cc:260] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.676167 14567 ts_tablet_manager.cc:1398] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:25.677373 13448 raft_consensus.cc:2454] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:25.677330 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b2f3557f96d4f33b02941d086de1391" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:25.677495 13319 raft_consensus.cc:3037] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.677732 13514 leader_election.cc:302] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:25.677781 14541 leader_election.cc:288] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:25.678014 14264 raft_consensus.cc:2781] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:25.679873 13319 raft_consensus.cc:2454] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:25.680500 14264 raft_consensus.cc:686] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:25.680727 14264 consensus_queue.cc:237] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:25.683162 13045 catalog_manager.cc:5484] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:25.693841 14499 raft_consensus.cc:482] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:25.693979 14499 raft_consensus.cc:504] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.694682 14499 leader_election.cc:288] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:25.695056 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1c998f38c049eda8d4ff28fb2eca29" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:25.695106 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1c998f38c049eda8d4ff28fb2eca29" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:25.695245 13580 raft_consensus.cc:2454] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:25.695304 13319 raft_consensus.cc:2454] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:25.695593 13120 leader_election.cc:302] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:25.695977 14499 raft_consensus.cc:2781] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:25.696096 14499 raft_consensus.cc:482] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:25.696193 14499 raft_consensus.cc:3037] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.698380 14499 raft_consensus.cc:504] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.698875 14499 leader_election.cc:288] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:25.699256 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1c998f38c049eda8d4ff28fb2eca29" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:25.699321 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad1c998f38c049eda8d4ff28fb2eca29" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:25.699421 13580 raft_consensus.cc:3037] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.699506 13319 raft_consensus.cc:3037] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.702075 13319 raft_consensus.cc:2454] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:25.702087 13580 raft_consensus.cc:2454] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:25.702387 13122 leader_election.cc:302] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:25.702709 14499 raft_consensus.cc:2781] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:25.702821 14499 raft_consensus.cc:686] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:25.703038 14499 consensus_queue.cc:237] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.705221 13045 catalog_manager.cc:5484] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:25.916349 14357 raft_consensus.cc:482] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:25.916558 14357 raft_consensus.cc:504] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.917165 14357 leader_election.cc:288] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:25.917698 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc40c95ebb94109a784c659b5b97ffe" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:25.917917 13448 raft_consensus.cc:2454] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:25.917847 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc40c95ebb94109a784c659b5b97ffe" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:25.918085 13186 raft_consensus.cc:2454] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:25.918314 13251 leader_election.cc:302] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:25.918583 14357 raft_consensus.cc:2781] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:25.918661 14357 raft_consensus.cc:482] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:25.918751 14357 raft_consensus.cc:3037] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.920583 14357 raft_consensus.cc:504] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.921113 14357 leader_election.cc:288] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:25.921474 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc40c95ebb94109a784c659b5b97ffe" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:25.921662 13448 raft_consensus.cc:3037] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.921617 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7cc40c95ebb94109a784c659b5b97ffe" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:25.921813 13186 raft_consensus.cc:3037] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:25.923288 13448 raft_consensus.cc:2454] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:25.923586 13186 raft_consensus.cc:2454] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:25.923614 13251 leader_election.cc:302] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:25.923949 14357 raft_consensus.cc:2781] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:25.924110 14357 raft_consensus.cc:686] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:25.924364 14357 consensus_queue.cc:237] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:25.927373 13039 catalog_manager.cc:5484] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:26.255182 14499 consensus_queue.cc:1035] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:26.260499 14264 consensus_queue.cc:1035] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:26.267753 14499 consensus_queue.cc:1035] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:26.268823 14264 consensus_queue.cc:1035] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:26.332073 14584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:26.332321 14584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240502 14:14:26.334954 14357 consensus_queue.cc:1035] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:26.339934 14603 consensus_queue.cc:1035] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:26.340927 14584 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 204.029 ms
time per row: 0.102014 ms
I20240502 14:14:26.688619 13045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:26.696247 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa51110016d4cafafbe25a1bd213f2e (DEFAULT_TABLE table=rebalance_test_table_5 [id=c97a45b4ee8c48a38ebd0709da5770f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.696557 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa51110016d4cafafbe25a1bd213f2e (DEFAULT_TABLE table=rebalance_test_table_5 [id=c97a45b4ee8c48a38ebd0709da5770f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.696682 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa51110016d4cafafbe25a1bd213f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.696987 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa51110016d4cafafbe25a1bd213f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.697185 13167 tablet_service.cc:1467] Processing CreateTablet for tablet e3be42ac139741d19dd872ab551bebae (DEFAULT_TABLE table=rebalance_test_table_5 [id=c97a45b4ee8c48a38ebd0709da5770f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.697574 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3be42ac139741d19dd872ab551bebae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.697763 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa51110016d4cafafbe25a1bd213f2e (DEFAULT_TABLE table=rebalance_test_table_5 [id=c97a45b4ee8c48a38ebd0709da5770f7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.698168 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa51110016d4cafafbe25a1bd213f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.698204 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 22705e1aee0947d8996575b93efd3f1f (DEFAULT_TABLE table=rebalance_test_table_5 [id=c97a45b4ee8c48a38ebd0709da5770f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.698596 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22705e1aee0947d8996575b93efd3f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.703332 13428 tablet_service.cc:1467] Processing CreateTablet for tablet 22705e1aee0947d8996575b93efd3f1f (DEFAULT_TABLE table=rebalance_test_table_5 [id=c97a45b4ee8c48a38ebd0709da5770f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.703656 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22705e1aee0947d8996575b93efd3f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.703766 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 22705e1aee0947d8996575b93efd3f1f (DEFAULT_TABLE table=rebalance_test_table_5 [id=c97a45b4ee8c48a38ebd0709da5770f7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.704090 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22705e1aee0947d8996575b93efd3f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.704085 13560 tablet_service.cc:1467] Processing CreateTablet for tablet e3be42ac139741d19dd872ab551bebae (DEFAULT_TABLE table=rebalance_test_table_5 [id=c97a45b4ee8c48a38ebd0709da5770f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.704396 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3be42ac139741d19dd872ab551bebae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.707331 13429 tablet_service.cc:1467] Processing CreateTablet for tablet e3be42ac139741d19dd872ab551bebae (DEFAULT_TABLE table=rebalance_test_table_5 [id=c97a45b4ee8c48a38ebd0709da5770f7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:26.707697 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3be42ac139741d19dd872ab551bebae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:26.720662 14652 tablet_bootstrap.cc:492] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:26.722440 14651 tablet_bootstrap.cc:492] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:26.722744 14652 tablet_bootstrap.cc:654] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.723372 14653 tablet_bootstrap.cc:492] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:26.724077 14650 tablet_bootstrap.cc:492] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:26.724516 14651 tablet_bootstrap.cc:654] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.725533 14653 tablet_bootstrap.cc:654] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.725533 14650 tablet_bootstrap.cc:654] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.728413 14652 tablet_bootstrap.cc:492] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:26.728456 14651 tablet_bootstrap.cc:492] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:26.728580 14652 ts_tablet_manager.cc:1367] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 14:14:26.728600 14651 ts_tablet_manager.cc:1367] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240502 14:14:26.729533 14651 raft_consensus.cc:348] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.729530 14652 raft_consensus.cc:348] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:26.729753 14651 raft_consensus.cc:374] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.729758 14652 raft_consensus.cc:374] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.729835 14651 raft_consensus.cc:729] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.729857 14652 raft_consensus.cc:729] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.730077 14651 consensus_queue.cc:260] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.730072 14652 consensus_queue.cc:260] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:26.730643 14652 ts_tablet_manager.cc:1398] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:26.731483 14651 ts_tablet_manager.cc:1398] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:26.731757 14651 tablet_bootstrap.cc:492] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:26.731897 14653 tablet_bootstrap.cc:492] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:26.732046 14653 ts_tablet_manager.cc:1367] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240502 14:14:26.732725 14653 raft_consensus.cc:348] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.732982 14653 raft_consensus.cc:374] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.733052 14651 tablet_bootstrap.cc:654] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.733131 14653 raft_consensus.cc:729] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.733425 14653 consensus_queue.cc:260] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.734059 14650 tablet_bootstrap.cc:492] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:26.734189 14650 ts_tablet_manager.cc:1367] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240502 14:14:26.734916 14650 raft_consensus.cc:348] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.735080 14650 raft_consensus.cc:374] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.735167 14650 raft_consensus.cc:729] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.735388 14650 consensus_queue.cc:260] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.735918 14650 ts_tablet_manager.cc:1398] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:26.736224 14650 tablet_bootstrap.cc:492] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:26.738209 14650 tablet_bootstrap.cc:654] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.736754 14653 ts_tablet_manager.cc:1398] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:26.743407 14653 tablet_bootstrap.cc:492] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:26.744191 14650 tablet_bootstrap.cc:492] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:26.744231 14651 tablet_bootstrap.cc:492] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:26.744343 14650 ts_tablet_manager.cc:1367] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.004s
I20240502 14:14:26.744521 14651 ts_tablet_manager.cc:1367] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20240502 14:14:26.744804 14653 tablet_bootstrap.cc:654] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.745077 14651 raft_consensus.cc:348] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.745254 14651 raft_consensus.cc:374] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.745169 14650 raft_consensus.cc:348] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:26.745329 14651 raft_consensus.cc:729] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.745378 14650 raft_consensus.cc:374] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.745448 14650 raft_consensus.cc:729] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.745590 14651 consensus_queue.cc:260] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.745741 14650 consensus_queue.cc:260] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:26.746248 14651 ts_tablet_manager.cc:1398] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:26.746495 14651 tablet_bootstrap.cc:492] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:26.747069 14653 tablet_bootstrap.cc:492] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:26.747109 14650 ts_tablet_manager.cc:1398] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:26.747300 14653 ts_tablet_manager.cc:1367] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 14:14:26.747972 14651 tablet_bootstrap.cc:654] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.748003 14653 raft_consensus.cc:348] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.748293 14653 raft_consensus.cc:374] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.748404 14653 raft_consensus.cc:729] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.748675 14653 consensus_queue.cc:260] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.749248 14653 ts_tablet_manager.cc:1398] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:26.749907 14650 tablet_bootstrap.cc:492] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:26.750859 14651 tablet_bootstrap.cc:492] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:26.751013 14651 ts_tablet_manager.cc:1367] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 14:14:26.751088 14650 tablet_bootstrap.cc:654] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:26.751632 14651 raft_consensus.cc:348] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:26.751837 14651 raft_consensus.cc:374] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.751940 14651 raft_consensus.cc:729] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.752166 14651 consensus_queue.cc:260] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:26.752645 14651 ts_tablet_manager.cc:1398] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:26.752929 14650 tablet_bootstrap.cc:492] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:26.753041 14650 ts_tablet_manager.cc:1367] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:14:26.753736 14650 raft_consensus.cc:348] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.753868 14650 raft_consensus.cc:374] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:26.753964 14650 raft_consensus.cc:729] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:26.754128 14650 consensus_queue.cc:260] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.754535 14650 ts_tablet_manager.cc:1398] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:14:26.779481 14586 raft_consensus.cc:482] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:26.779587 14586 raft_consensus.cc:504] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:26.780143 14586 leader_election.cc:288] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:26.780505 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22705e1aee0947d8996575b93efd3f1f" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:26.780583 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22705e1aee0947d8996575b93efd3f1f" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:26.780743 13319 raft_consensus.cc:2454] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:26.780750 13448 raft_consensus.cc:2454] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:26.781152 13122 leader_election.cc:302] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:26.781389 14586 raft_consensus.cc:2781] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:26.781474 14586 raft_consensus.cc:482] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:26.781529 14586 raft_consensus.cc:3037] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.783035 14586 raft_consensus.cc:504] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:26.783514 14586 leader_election.cc:288] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:26.783882 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22705e1aee0947d8996575b93efd3f1f" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:26.783946 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22705e1aee0947d8996575b93efd3f1f" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:26.784063 13319 raft_consensus.cc:3037] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.784111 13448 raft_consensus.cc:3037] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.785823 13448 raft_consensus.cc:2454] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:26.785847 13319 raft_consensus.cc:2454] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:26.786202 13120 leader_election.cc:302] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:26.786494 14586 raft_consensus.cc:2781] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:26.786654 14586 raft_consensus.cc:686] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:26.786962 14586 consensus_queue.cc:237] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:26.789588 13042 catalog_manager.cc:5484] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:26.833516 14518 raft_consensus.cc:482] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:26.833669 14518 raft_consensus.cc:504] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.834107 14518 leader_election.cc:288] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:26.834497 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3be42ac139741d19dd872ab551bebae" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:26.834579 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3be42ac139741d19dd872ab551bebae" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:26.834726 13186 raft_consensus.cc:2454] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:26.834815 13580 raft_consensus.cc:2454] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:26.835162 13382 leader_election.cc:302] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:26.835433 14518 raft_consensus.cc:2781] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:26.835518 14518 raft_consensus.cc:482] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:26.835626 14518 raft_consensus.cc:3037] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.837925 14518 raft_consensus.cc:504] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.838436 14518 leader_election.cc:288] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:26.838775 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3be42ac139741d19dd872ab551bebae" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:26.838887 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3be42ac139741d19dd872ab551bebae" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:26.838946 13186 raft_consensus.cc:3037] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.839075 13580 raft_consensus.cc:3037] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.840644 13186 raft_consensus.cc:2454] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:26.840939 13580 raft_consensus.cc:2454] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:26.840997 13382 leader_election.cc:302] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:26.841320 14518 raft_consensus.cc:2781] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:26.841481 14518 raft_consensus.cc:686] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:26.841725 14518 consensus_queue.cc:237] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.844336 13042 catalog_manager.cc:5484] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:26.976538 14264 raft_consensus.cc:482] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:26.976702 14264 raft_consensus.cc:504] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.977360 14264 leader_election.cc:288] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:26.977811 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa51110016d4cafafbe25a1bd213f2e" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:26.977864 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa51110016d4cafafbe25a1bd213f2e" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:26.978031 13448 raft_consensus.cc:2454] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:26.978044 13186 raft_consensus.cc:2454] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:26.978374 13514 leader_election.cc:302] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:26.978631 14264 raft_consensus.cc:2781] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:26.978744 14264 raft_consensus.cc:482] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:26.978822 14264 raft_consensus.cc:3037] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.980787 14264 raft_consensus.cc:504] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.981252 14264 leader_election.cc:288] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:26.981643 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa51110016d4cafafbe25a1bd213f2e" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:26.981694 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fa51110016d4cafafbe25a1bd213f2e" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:26.981842 13448 raft_consensus.cc:3037] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.981873 13186 raft_consensus.cc:3037] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:26.983441 13186 raft_consensus.cc:2454] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:26.983742 13514 leader_election.cc:302] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:26.983930 14264 raft_consensus.cc:2781] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:26.984045 14264 raft_consensus.cc:686] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:26.984295 14264 consensus_queue.cc:237] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:26.986676 13042 catalog_manager.cc:5484] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:27.002774 13448 raft_consensus.cc:2454] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:27.224536 14518 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:27.230697 14518 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:27.319080 14667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:27.319425 14667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:27.326864 14667 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 14:14:27.358366 14586 consensus_queue.cc:1035] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:27.370975 14690 consensus_queue.cc:1035] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:27.381712 14264 consensus_queue.cc:1035] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:27.386634 14594 consensus_queue.cc:1035] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 237.343 ms
time per row: 0.118671 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:14:28.596446 13045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:28.606005 13298 tablet_service.cc:1467] Processing CreateTablet for tablet c1edaea0b866430ba8c5bd6bac5a9b66 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.606422 13297 tablet_service.cc:1467] Processing CreateTablet for tablet c54b3251d0ff4e48b93b95634b589b4c (DEFAULT_TABLE table=rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.606604 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 8e96c49666584fceb759383d24ac1c4a (DEFAULT_TABLE table=rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.607707 13427 tablet_service.cc:1467] Processing CreateTablet for tablet c54b3251d0ff4e48b93b95634b589b4c (DEFAULT_TABLE table=rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.608132 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54b3251d0ff4e48b93b95634b589b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.608168 13559 tablet_service.cc:1467] Processing CreateTablet for tablet c54b3251d0ff4e48b93b95634b589b4c (DEFAULT_TABLE table=rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.608222 13561 tablet_service.cc:1467] Processing CreateTablet for tablet 8e96c49666584fceb759383d24ac1c4a (DEFAULT_TABLE table=rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.608546 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54b3251d0ff4e48b93b95634b589b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.608806 13167 tablet_service.cc:1467] Processing CreateTablet for tablet c1edaea0b866430ba8c5bd6bac5a9b66 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.609158 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1edaea0b866430ba8c5bd6bac5a9b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.612124 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e96c49666584fceb759383d24ac1c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.613734 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 8e96c49666584fceb759383d24ac1c4a (DEFAULT_TABLE table=rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.614177 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e96c49666584fceb759383d24ac1c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.617892 13560 tablet_service.cc:1467] Processing CreateTablet for tablet c1edaea0b866430ba8c5bd6bac5a9b66 (DEFAULT_TABLE table=rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:28.618295 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1edaea0b866430ba8c5bd6bac5a9b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.618405 14739 tablet_bootstrap.cc:492] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:28.620007 14739 tablet_bootstrap.cc:654] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.606514 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1edaea0b866430ba8c5bd6bac5a9b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.622730 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e96c49666584fceb759383d24ac1c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.623262 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c54b3251d0ff4e48b93b95634b589b4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:28.628257 14739 tablet_bootstrap.cc:492] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:28.628432 14739 ts_tablet_manager.cc:1367] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 14:14:28.629164 14739 raft_consensus.cc:348] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:28.629877 14739 raft_consensus.cc:374] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.630007 14739 raft_consensus.cc:729] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.630296 14739 consensus_queue.cc:260] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:28.633770 14739 ts_tablet_manager.cc:1398] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:14:28.638425 14738 tablet_bootstrap.cc:492] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:28.639402 14743 tablet_bootstrap.cc:492] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:28.639777 14742 tablet_bootstrap.cc:492] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:28.640563 14738 tablet_bootstrap.cc:654] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.641647 14743 tablet_bootstrap.cc:654] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.642442 14738 tablet_bootstrap.cc:492] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:28.642632 14738 ts_tablet_manager.cc:1367] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:14:28.642649 14742 tablet_bootstrap.cc:654] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.643316 14743 tablet_bootstrap.cc:492] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:28.643322 14738 raft_consensus.cc:348] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:28.643489 14743 ts_tablet_manager.cc:1367] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:28.643592 14738 raft_consensus.cc:374] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.643726 14738 raft_consensus.cc:729] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.644189 14743 raft_consensus.cc:348] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:28.644358 14743 raft_consensus.cc:374] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.644446 14743 raft_consensus.cc:729] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.644639 14743 consensus_queue.cc:260] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:28.644953 14738 consensus_queue.cc:260] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:28.645155 14743 ts_tablet_manager.cc:1398] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:28.645498 14743 tablet_bootstrap.cc:492] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:28.645639 14738 ts_tablet_manager.cc:1398] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:28.645910 14738 tablet_bootstrap.cc:492] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:28.647167 14743 tablet_bootstrap.cc:654] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.649881 14738 tablet_bootstrap.cc:654] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.650759 14742 tablet_bootstrap.cc:492] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:28.650882 14742 ts_tablet_manager.cc:1367] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20240502 14:14:28.651605 14742 raft_consensus.cc:348] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:28.651811 14742 raft_consensus.cc:374] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.651939 14742 raft_consensus.cc:729] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.652223 14742 consensus_queue.cc:260] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:28.652549 14743 tablet_bootstrap.cc:492] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:28.652724 14743 ts_tablet_manager.cc:1367] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240502 14:14:28.652788 14742 ts_tablet_manager.cc:1398] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:28.653033 14738 tablet_bootstrap.cc:492] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:28.653391 14742 tablet_bootstrap.cc:492] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:28.653452 14743 raft_consensus.cc:348] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:28.653641 14743 raft_consensus.cc:374] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.653717 14743 raft_consensus.cc:729] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.653906 14743 consensus_queue.cc:260] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:28.654395 14743 ts_tablet_manager.cc:1398] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:28.654685 14738 ts_tablet_manager.cc:1367] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 14:14:28.654975 14742 tablet_bootstrap.cc:654] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.656731 14742 tablet_bootstrap.cc:492] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:28.656931 14742 ts_tablet_manager.cc:1367] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 14:14:28.657248 14738 raft_consensus.cc:348] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:28.657526 14738 raft_consensus.cc:374] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.657608 14738 raft_consensus.cc:729] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.657768 14742 raft_consensus.cc:348] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:28.658000 14742 raft_consensus.cc:374] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.658147 14742 raft_consensus.cc:729] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.658403 14742 consensus_queue.cc:260] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:28.658092 14738 consensus_queue.cc:260] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:28.659035 14742 ts_tablet_manager.cc:1398] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:28.659301 14742 tablet_bootstrap.cc:492] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:28.660188 14738 ts_tablet_manager.cc:1398] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:14:28.660624 14738 tablet_bootstrap.cc:492] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:28.660832 14742 tablet_bootstrap.cc:654] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.662827 14738 tablet_bootstrap.cc:654] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:28.664348 14738 tablet_bootstrap.cc:492] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:28.664443 14738 ts_tablet_manager.cc:1367] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:14:28.664891 14738 raft_consensus.cc:348] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:28.665014 14738 raft_consensus.cc:374] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.665094 14738 raft_consensus.cc:729] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.665294 14738 consensus_queue.cc:260] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:28.665702 14738 ts_tablet_manager.cc:1398] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:14:28.670051 14742 tablet_bootstrap.cc:492] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:28.670162 14742 ts_tablet_manager.cc:1367] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 14:14:28.670704 14742 raft_consensus.cc:348] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:28.670882 14742 raft_consensus.cc:374] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:28.670954 14742 raft_consensus.cc:729] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:28.671157 14742 consensus_queue.cc:260] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:28.671591 14742 ts_tablet_manager.cc:1398] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:14:28.694576 14712 raft_consensus.cc:482] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:28.694693 14712 raft_consensus.cc:504] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:28.695209 14712 leader_election.cc:288] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:28.695591 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1edaea0b866430ba8c5bd6bac5a9b66" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:28.695585 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1edaea0b866430ba8c5bd6bac5a9b66" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:28.695786 13318 raft_consensus.cc:2454] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:28.695786 13188 raft_consensus.cc:2454] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:28.696183 13516 leader_election.cc:302] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:28.696476 14712 raft_consensus.cc:2781] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:28.696570 14712 raft_consensus.cc:482] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:28.696656 14712 raft_consensus.cc:3037] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.698395 14712 raft_consensus.cc:504] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:28.698868 14712 leader_election.cc:288] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:28.699201 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1edaea0b866430ba8c5bd6bac5a9b66" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:28.699242 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1edaea0b866430ba8c5bd6bac5a9b66" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:28.699380 13188 raft_consensus.cc:3037] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.699380 13318 raft_consensus.cc:3037] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.701922 13318 raft_consensus.cc:2454] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:28.701922 13188 raft_consensus.cc:2454] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:28.702224 13514 leader_election.cc:302] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:28.702443 14712 raft_consensus.cc:2781] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:28.702550 14712 raft_consensus.cc:686] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:28.702800 14712 consensus_queue.cc:237] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:28.705029 13042 catalog_manager.cc:5484] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:28.754779 14731 raft_consensus.cc:482] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:28.754948 14731 raft_consensus.cc:504] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:28.755432 14731 leader_election.cc:288] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:28.755858 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54b3251d0ff4e48b93b95634b589b4c" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:28.755908 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54b3251d0ff4e48b93b95634b589b4c" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:28.756083 13580 raft_consensus.cc:2454] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:28.756094 13448 raft_consensus.cc:2454] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:28.756507 13251 leader_election.cc:302] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:28.756773 14731 raft_consensus.cc:2781] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:28.756892 14731 raft_consensus.cc:482] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:28.756983 14731 raft_consensus.cc:3037] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.758684 14731 raft_consensus.cc:504] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:28.759126 14731 leader_election.cc:288] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:28.759536 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54b3251d0ff4e48b93b95634b589b4c" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:28.759600 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c54b3251d0ff4e48b93b95634b589b4c" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:28.759713 13580 raft_consensus.cc:3037] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.759759 13448 raft_consensus.cc:3037] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.762389 13448 raft_consensus.cc:2454] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:28.762387 13580 raft_consensus.cc:2454] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:28.762745 13251 leader_election.cc:302] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:28.763001 14731 raft_consensus.cc:2781] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:28.763166 14731 raft_consensus.cc:686] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:28.763422 14731 consensus_queue.cc:237] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:28.765818 13042 catalog_manager.cc:5484] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:28.801506 14712 raft_consensus.cc:482] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:28.801647 14712 raft_consensus.cc:504] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:28.802096 14712 leader_election.cc:288] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:28.802431 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e96c49666584fceb759383d24ac1c4a" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:28.802517 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e96c49666584fceb759383d24ac1c4a" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:28.802635 13318 raft_consensus.cc:2454] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:28.802762 13188 raft_consensus.cc:2454] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:28.802953 13516 leader_election.cc:302] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:28.803190 14712 raft_consensus.cc:2781] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:28.803272 14712 raft_consensus.cc:482] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:28.803365 14712 raft_consensus.cc:3037] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.804926 14712 raft_consensus.cc:504] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:28.805472 14712 leader_election.cc:288] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:28.805739 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e96c49666584fceb759383d24ac1c4a" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:28.805927 13318 raft_consensus.cc:3037] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.805928 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e96c49666584fceb759383d24ac1c4a" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:28.806118 13188 raft_consensus.cc:3037] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:28.808630 13318 raft_consensus.cc:2454] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:28.808890 13188 raft_consensus.cc:2454] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:28.809021 13516 leader_election.cc:302] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:28.809268 14712 raft_consensus.cc:2781] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:28.809405 14712 raft_consensus.cc:686] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:28.809618 14712 consensus_queue.cc:237] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:28.811769 13042 catalog_manager.cc:5484] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:29.117832 14755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:29.118084 14755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:29.124958 14755 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:29.143441 14731 consensus_queue.cc:1035] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:29.151152 14731 consensus_queue.cc:1035] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240502 14:14:29.193863 14712 consensus_queue.cc:1035] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:29.200565 14785 consensus_queue.cc:1035] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:29.250208 13188 raft_consensus.cc:1261] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:29.250849 14785 consensus_queue.cc:1035] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:29.252305 13318 raft_consensus.cc:1261] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:29.254344 14712 consensus_queue.cc:1035] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:29.361600 13251 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.196:46817, user_credentials={real_user=slave}} blocked reactor thread for 55385.9us
INSERT report
rows total: 2000
time total: 187.976 ms
time per row: 0.093988 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:14:30.306699 13045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:30.317139 13167 tablet_service.cc:1467] Processing CreateTablet for tablet ae45cbb45edf41808686de3418231730 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fe675f309a11456aaf1fc329cfdb1707]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.317490 13427 tablet_service.cc:1467] Processing CreateTablet for tablet ae45cbb45edf41808686de3418231730 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fe675f309a11456aaf1fc329cfdb1707]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.317667 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae45cbb45edf41808686de3418231730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.317888 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae45cbb45edf41808686de3418231730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.321081 13297 tablet_service.cc:1467] Processing CreateTablet for tablet ae45cbb45edf41808686de3418231730 (DEFAULT_TABLE table=rebalance_test_table_7 [id=fe675f309a11456aaf1fc329cfdb1707]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.321535 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae45cbb45edf41808686de3418231730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.321090 13298 tablet_service.cc:1467] Processing CreateTablet for tablet dee785c61c73426fb5665c1334ff372b (DEFAULT_TABLE table=rebalance_test_table_7 [id=fe675f309a11456aaf1fc329cfdb1707]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.321830 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee785c61c73426fb5665c1334ff372b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.323057 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 16c817b864c4425eaa85ac4b5aff59fa (DEFAULT_TABLE table=rebalance_test_table_7 [id=fe675f309a11456aaf1fc329cfdb1707]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.323132 13560 tablet_service.cc:1467] Processing CreateTablet for tablet dee785c61c73426fb5665c1334ff372b (DEFAULT_TABLE table=rebalance_test_table_7 [id=fe675f309a11456aaf1fc329cfdb1707]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.323488 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c817b864c4425eaa85ac4b5aff59fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.323940 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 16c817b864c4425eaa85ac4b5aff59fa (DEFAULT_TABLE table=rebalance_test_table_7 [id=fe675f309a11456aaf1fc329cfdb1707]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.324258 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c817b864c4425eaa85ac4b5aff59fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.328472 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 16c817b864c4425eaa85ac4b5aff59fa (DEFAULT_TABLE table=rebalance_test_table_7 [id=fe675f309a11456aaf1fc329cfdb1707]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.328887 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c817b864c4425eaa85ac4b5aff59fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.332437 14830 tablet_bootstrap.cc:492] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:30.332760 13428 tablet_service.cc:1467] Processing CreateTablet for tablet dee785c61c73426fb5665c1334ff372b (DEFAULT_TABLE table=rebalance_test_table_7 [id=fe675f309a11456aaf1fc329cfdb1707]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:30.333159 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee785c61c73426fb5665c1334ff372b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.334141 14830 tablet_bootstrap.cc:654] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.337155 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee785c61c73426fb5665c1334ff372b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:30.346566 14834 tablet_bootstrap.cc:492] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:30.349507 14834 tablet_bootstrap.cc:654] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.358295 14830 tablet_bootstrap.cc:492] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:30.358546 14830 ts_tablet_manager.cc:1367] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.025s
I20240502 14:14:30.359560 14830 raft_consensus.cc:348] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:30.359917 14830 raft_consensus.cc:374] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.360083 14830 raft_consensus.cc:729] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.360404 14830 consensus_queue.cc:260] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:30.361136 14830 ts_tablet_manager.cc:1398] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:30.361459 14830 tablet_bootstrap.cc:492] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:30.363006 14830 tablet_bootstrap.cc:654] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.371376 14824 raft_consensus.cc:482] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:30.371516 14824 raft_consensus.cc:504] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:30.372138 14824 leader_election.cc:288] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:30.372980 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae45cbb45edf41808686de3418231730" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:30.373437 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae45cbb45edf41808686de3418231730" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
W20240502 14:14:30.373950 13122 leader_election.cc:341] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:30.374099 14837 tablet_bootstrap.cc:492] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
W20240502 14:14:30.374264 13120 leader_election.cc:341] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:30.374374 13120 leader_election.cc:302] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d
I20240502 14:14:30.374744 14824 raft_consensus.cc:2726] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:30.376518 14837 tablet_bootstrap.cc:654] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
W20240502 14:14:30.380633 13251 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.193:36687, user_credentials={real_user=slave}} blocked reactor thread for 52415.7us
I20240502 14:14:30.385324 14831 tablet_bootstrap.cc:492] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:30.385411 14834 tablet_bootstrap.cc:492] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:30.385577 14834 ts_tablet_manager.cc:1367] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.039s user 0.009s sys 0.016s
I20240502 14:14:30.386183 14834 raft_consensus.cc:348] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:30.386386 14834 raft_consensus.cc:374] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.386512 14834 raft_consensus.cc:729] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.386760 14834 consensus_queue.cc:260] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:30.387292 14831 tablet_bootstrap.cc:654] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.387418 14834 ts_tablet_manager.cc:1398] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:30.387676 14834 tablet_bootstrap.cc:492] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:30.390928 14834 tablet_bootstrap.cc:654] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.391932 14830 tablet_bootstrap.cc:492] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:30.392138 14830 ts_tablet_manager.cc:1367] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.031s user 0.000s sys 0.009s
I20240502 14:14:30.393062 14830 raft_consensus.cc:348] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:30.393378 14830 raft_consensus.cc:374] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.393497 14830 raft_consensus.cc:729] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.393841 14830 consensus_queue.cc:260] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:30.394562 14830 ts_tablet_manager.cc:1398] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:30.395871 14831 tablet_bootstrap.cc:492] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:30.396030 14831 ts_tablet_manager.cc:1367] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240502 14:14:30.396842 14837 tablet_bootstrap.cc:492] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:30.396705 14831 raft_consensus.cc:348] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:30.396971 14831 raft_consensus.cc:374] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.397033 14837 ts_tablet_manager.cc:1367] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.011s
I20240502 14:14:30.397133 14831 raft_consensus.cc:729] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.397406 14831 consensus_queue.cc:260] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:30.397825 14837 raft_consensus.cc:348] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:30.398037 14837 raft_consensus.cc:374] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.398152 14837 raft_consensus.cc:729] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.398203 14831 ts_tablet_manager.cc:1398] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:30.398586 14831 tablet_bootstrap.cc:492] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:30.398391 14837 consensus_queue.cc:260] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:30.399060 14837 ts_tablet_manager.cc:1398] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:30.399398 14837 tablet_bootstrap.cc:492] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:30.400815 14831 tablet_bootstrap.cc:654] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.400910 14837 tablet_bootstrap.cc:654] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.402643 14837 tablet_bootstrap.cc:492] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:30.402848 14837 ts_tablet_manager.cc:1367] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:30.403584 14837 raft_consensus.cc:348] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:30.403836 14837 raft_consensus.cc:374] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.403955 14837 raft_consensus.cc:729] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.404175 14834 tablet_bootstrap.cc:492] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:30.404211 14837 consensus_queue.cc:260] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:30.404383 14831 tablet_bootstrap.cc:492] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:30.404546 14831 ts_tablet_manager.cc:1367] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240502 14:14:30.404723 14837 ts_tablet_manager.cc:1398] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:30.405004 14837 tablet_bootstrap.cc:492] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:30.404397 14834 ts_tablet_manager.cc:1367] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.004s
I20240502 14:14:30.405262 14831 raft_consensus.cc:348] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:30.405489 14831 raft_consensus.cc:374] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.405599 14831 raft_consensus.cc:729] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.405879 14831 consensus_queue.cc:260] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:30.405941 14834 raft_consensus.cc:348] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:30.406389 14837 tablet_bootstrap.cc:654] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:30.406580 14831 ts_tablet_manager.cc:1398] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:30.407955 14837 tablet_bootstrap.cc:492] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:30.408170 14837 ts_tablet_manager.cc:1367] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:30.408627 14834 raft_consensus.cc:374] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.408726 14834 raft_consensus.cc:729] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.408732 14837 raft_consensus.cc:348] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:30.408936 14837 raft_consensus.cc:374] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:30.409044 14837 raft_consensus.cc:729] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:30.409215 14837 consensus_queue.cc:260] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:30.409608 14837 ts_tablet_manager.cc:1398] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:14:30.408926 14834 consensus_queue.cc:260] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:30.412159 14834 ts_tablet_manager.cc:1398] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 14:14:30.529696 14823 raft_consensus.cc:482] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:30.529821 14823 raft_consensus.cc:504] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:30.530263 14823 leader_election.cc:288] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:30.530583 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c817b864c4425eaa85ac4b5aff59fa" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:30.530640 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c817b864c4425eaa85ac4b5aff59fa" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:30.530738 13448 raft_consensus.cc:2454] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:30.530872 13188 raft_consensus.cc:2454] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:30.531126 13514 leader_election.cc:302] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:30.531361 14823 raft_consensus.cc:2781] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:30.531450 14823 raft_consensus.cc:482] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:30.531525 14823 raft_consensus.cc:3037] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.533111 14823 raft_consensus.cc:504] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:30.533576 14823 leader_election.cc:288] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:30.533947 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c817b864c4425eaa85ac4b5aff59fa" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:30.533994 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c817b864c4425eaa85ac4b5aff59fa" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:30.534103 13448 raft_consensus.cc:3037] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.534137 13188 raft_consensus.cc:3037] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.535677 13188 raft_consensus.cc:2454] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:30.535852 13448 raft_consensus.cc:2454] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:30.535970 13514 leader_election.cc:302] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:30.536187 14823 raft_consensus.cc:2781] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:30.536327 14823 raft_consensus.cc:686] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:30.536576 14823 consensus_queue.cc:237] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:30.538995 13036 catalog_manager.cc:5484] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:30.543905 14812 raft_consensus.cc:482] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:30.544031 14812 raft_consensus.cc:504] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:30.544483 14812 leader_election.cc:288] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:30.544778 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae45cbb45edf41808686de3418231730" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:30.544829 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae45cbb45edf41808686de3418231730" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:30.544925 13188 raft_consensus.cc:2454] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:30.544992 13448 raft_consensus.cc:2454] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:30.545320 13251 leader_election.cc:302] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:30.545522 14812 raft_consensus.cc:2781] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:30.545637 14812 raft_consensus.cc:482] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:30.545727 14812 raft_consensus.cc:3037] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.548034 14812 raft_consensus.cc:504] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:30.548583 14812 leader_election.cc:288] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:30.548869 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae45cbb45edf41808686de3418231730" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:30.548975 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae45cbb45edf41808686de3418231730" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:30.549127 13188 raft_consensus.cc:3037] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.549151 13448 raft_consensus.cc:3037] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.550760 13448 raft_consensus.cc:2454] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:30.550817 13188 raft_consensus.cc:2454] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:30.551069 13251 leader_election.cc:302] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:30.551281 14812 raft_consensus.cc:2781] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:30.551443 14812 raft_consensus.cc:686] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:30.551666 14812 consensus_queue.cc:237] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:30.553910 13036 catalog_manager.cc:5484] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:30.612675 14741 raft_consensus.cc:482] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:30.612815 14741 raft_consensus.cc:504] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:30.613332 14741 leader_election.cc:288] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:30.613618 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee785c61c73426fb5665c1334ff372b" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:30.613726 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee785c61c73426fb5665c1334ff372b" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:30.613860 13317 raft_consensus.cc:2454] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:30.613934 13580 raft_consensus.cc:2454] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:30.614215 13384 leader_election.cc:302] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:30.614452 14741 raft_consensus.cc:2781] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:30.614540 14741 raft_consensus.cc:482] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:30.614599 14741 raft_consensus.cc:3037] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.616236 14741 raft_consensus.cc:504] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:30.616719 14741 leader_election.cc:288] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:30.617031 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee785c61c73426fb5665c1334ff372b" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:30.617139 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dee785c61c73426fb5665c1334ff372b" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:30.617209 13317 raft_consensus.cc:3037] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.617354 13580 raft_consensus.cc:3037] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:30.618997 13317 raft_consensus.cc:2454] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:30.619096 13580 raft_consensus.cc:2454] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:30.619339 13384 leader_election.cc:302] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:30.619690 14741 raft_consensus.cc:2781] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:30.620062 14741 raft_consensus.cc:686] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:30.620344 14741 consensus_queue.cc:237] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:30.622787 13036 catalog_manager.cc:5484] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:30.983376 14823 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:30.989025 14823 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:31.001814 14850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20240502 14:14:31.001960 14812 consensus_queue.cc:1035] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:31.002142 14850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:31.011534 14850 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:31.015437 14832 consensus_queue.cc:1035] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240502 14:14:31.051728 14741 consensus_queue.cc:1035] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:31.057452 14741 consensus_queue.cc:1035] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 229.629 ms
time per row: 0.114814 ms
I20240502 14:14:31.379228 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:31.387243 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef68f6c2e4245dd86895c93ec270392 (DEFAULT_TABLE table=rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.387688 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ef68f6c2e4245dd86895c93ec270392. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.387636 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef68f6c2e4245dd86895c93ec270392 (DEFAULT_TABLE table=rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.388043 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ef68f6c2e4245dd86895c93ec270392. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.387992 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef68f6c2e4245dd86895c93ec270392 (DEFAULT_TABLE table=rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.388348 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ef68f6c2e4245dd86895c93ec270392. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.389134 13560 tablet_service.cc:1467] Processing CreateTablet for tablet dcc6274abeda40d78644d1685fa44ca6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.389524 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc6274abeda40d78644d1685fa44ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.393642 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdabe4a87ec4398b6e4508ccc0ae165 (DEFAULT_TABLE table=rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.393636 13297 tablet_service.cc:1467] Processing CreateTablet for tablet dcc6274abeda40d78644d1685fa44ca6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.394026 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bdabe4a87ec4398b6e4508ccc0ae165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.394064 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc6274abeda40d78644d1685fa44ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.399549 13167 tablet_service.cc:1467] Processing CreateTablet for tablet dcc6274abeda40d78644d1685fa44ca6 (DEFAULT_TABLE table=rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.400061 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcc6274abeda40d78644d1685fa44ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.399516 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdabe4a87ec4398b6e4508ccc0ae165 (DEFAULT_TABLE table=rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.402446 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bdabe4a87ec4398b6e4508ccc0ae165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.407769 13428 tablet_service.cc:1467] Processing CreateTablet for tablet 7bdabe4a87ec4398b6e4508ccc0ae165 (DEFAULT_TABLE table=rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:31.408149 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7bdabe4a87ec4398b6e4508ccc0ae165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:31.412778 14913 tablet_bootstrap.cc:492] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:31.415163 14913 tablet_bootstrap.cc:654] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.416457 14911 tablet_bootstrap.cc:492] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:31.417387 14914 tablet_bootstrap.cc:492] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:31.418457 14911 tablet_bootstrap.cc:654] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.419082 14914 tablet_bootstrap.cc:654] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.420845 14912 tablet_bootstrap.cc:492] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:31.421909 14913 tablet_bootstrap.cc:492] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:31.422125 14913 ts_tablet_manager.cc:1367] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:14:31.423123 14912 tablet_bootstrap.cc:654] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.423060 14913 raft_consensus.cc:348] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:31.423435 14913 raft_consensus.cc:374] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.423566 14913 raft_consensus.cc:729] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.423884 14913 consensus_queue.cc:260] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:31.424253 14911 tablet_bootstrap.cc:492] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:31.424422 14911 ts_tablet_manager.cc:1367] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 14:14:31.424610 14913 ts_tablet_manager.cc:1398] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:31.425246 14913 tablet_bootstrap.cc:492] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:31.425141 14911 raft_consensus.cc:348] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:31.425381 14911 raft_consensus.cc:374] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.425503 14911 raft_consensus.cc:729] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.425750 14911 consensus_queue.cc:260] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:31.426357 14911 ts_tablet_manager.cc:1398] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:31.427368 14911 tablet_bootstrap.cc:492] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:31.427969 14913 tablet_bootstrap.cc:654] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.432214 14911 tablet_bootstrap.cc:654] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.434695 14911 tablet_bootstrap.cc:492] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:31.434835 14912 tablet_bootstrap.cc:492] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:31.434870 14911 ts_tablet_manager.cc:1367] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240502 14:14:31.434963 14912 ts_tablet_manager.cc:1367] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.001s
I20240502 14:14:31.435497 14912 raft_consensus.cc:348] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:31.435642 14912 raft_consensus.cc:374] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.435737 14912 raft_consensus.cc:729] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.435781 14911 raft_consensus.cc:348] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:31.436017 14911 raft_consensus.cc:374] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.435935 14912 consensus_queue.cc:260] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:31.436105 14911 raft_consensus.cc:729] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.436342 14911 consensus_queue.cc:260] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:31.436614 14912 ts_tablet_manager.cc:1398] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:31.436865 14911 ts_tablet_manager.cc:1398] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:31.436894 14913 tablet_bootstrap.cc:492] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:31.436882 14912 tablet_bootstrap.cc:492] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:31.437206 14911 tablet_bootstrap.cc:492] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:31.437690 14913 ts_tablet_manager.cc:1367] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20240502 14:14:31.438690 14911 tablet_bootstrap.cc:654] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.438654 14913 raft_consensus.cc:348] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:31.438889 14913 raft_consensus.cc:374] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.439000 14913 raft_consensus.cc:729] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.439282 14913 consensus_queue.cc:260] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:31.441763 14913 ts_tablet_manager.cc:1398] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:14:31.442905 14911 tablet_bootstrap.cc:492] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:31.443024 14911 ts_tablet_manager.cc:1367] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20240502 14:14:31.443646 14911 raft_consensus.cc:348] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:31.443869 14911 raft_consensus.cc:374] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.443953 14912 tablet_bootstrap.cc:654] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.444003 14911 raft_consensus.cc:729] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.444193 14911 consensus_queue.cc:260] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:31.444676 14911 ts_tablet_manager.cc:1398] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:31.445401 14914 tablet_bootstrap.cc:492] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:31.445544 14914 ts_tablet_manager.cc:1367] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.010s
I20240502 14:14:31.445645 14912 tablet_bootstrap.cc:492] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:31.445781 14912 ts_tablet_manager.cc:1367] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240502 14:14:31.446242 14914 raft_consensus.cc:348] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:31.446432 14914 raft_consensus.cc:374] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.446380 14912 raft_consensus.cc:348] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:31.446568 14914 raft_consensus.cc:729] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.446591 14912 raft_consensus.cc:374] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.446681 14912 raft_consensus.cc:729] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.446843 14914 consensus_queue.cc:260] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:31.446931 14912 consensus_queue.cc:260] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:31.447428 14914 ts_tablet_manager.cc:1398] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:31.447499 14912 ts_tablet_manager.cc:1398] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:31.447751 14914 tablet_bootstrap.cc:492] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:31.448666 14823 raft_consensus.cc:482] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:31.448804 14823 raft_consensus.cc:504] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:31.449149 14914 tablet_bootstrap.cc:654] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:31.449442 14823 leader_election.cc:288] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:31.449992 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc6274abeda40d78644d1685fa44ca6" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:31.450209 13188 raft_consensus.cc:2454] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:31.450253 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc6274abeda40d78644d1685fa44ca6" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:31.450625 13514 leader_election.cc:302] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:31.451005 14823 raft_consensus.cc:2781] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240502 14:14:31.451071 13516 leader_election.cc:341] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:31.451108 14823 raft_consensus.cc:482] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:31.451193 14823 raft_consensus.cc:3037] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.453569 14823 raft_consensus.cc:504] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:31.454562 14823 leader_election.cc:288] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:31.454682 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc6274abeda40d78644d1685fa44ca6" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:31.455041 14914 tablet_bootstrap.cc:492] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:31.455188 14914 ts_tablet_manager.cc:1367] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
W20240502 14:14:31.455348 13516 leader_election.cc:341] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:31.455351 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcc6274abeda40d78644d1685fa44ca6" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:31.455539 13188 raft_consensus.cc:3037] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.455915 14914 raft_consensus.cc:348] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:31.456076 14914 raft_consensus.cc:374] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:31.456130 14914 raft_consensus.cc:729] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:31.456341 14914 consensus_queue.cc:260] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:31.456887 14914 ts_tablet_manager.cc:1398] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:31.457793 13188 raft_consensus.cc:2454] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:31.458097 13514 leader_election.cc:302] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters: cd67493f6f0142f896010f2b8a39887d
I20240502 14:14:31.458338 14823 raft_consensus.cc:2781] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:31.458490 14823 raft_consensus.cc:686] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:31.458700 14823 consensus_queue.cc:237] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:31.461181 13036 catalog_manager.cc:5484] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:31.516861 14897 raft_consensus.cc:482] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:31.517050 14897 raft_consensus.cc:504] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:31.517827 14897 leader_election.cc:288] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:31.518174 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ef68f6c2e4245dd86895c93ec270392" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:31.518373 13317 raft_consensus.cc:2454] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:31.518314 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ef68f6c2e4245dd86895c93ec270392" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:31.518507 13449 raft_consensus.cc:2454] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:31.518677 13122 leader_election.cc:302] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:31.518932 14897 raft_consensus.cc:2781] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:31.518970 14823 raft_consensus.cc:482] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:31.519043 14897 raft_consensus.cc:482] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:31.519133 14897 raft_consensus.cc:3037] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.519136 14823 raft_consensus.cc:504] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:31.519718 14823 leader_election.cc:288] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:31.520277 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdabe4a87ec4398b6e4508ccc0ae165" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:31.520365 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdabe4a87ec4398b6e4508ccc0ae165" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:31.520511 13188 raft_consensus.cc:2454] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:31.520558 13449 raft_consensus.cc:2454] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:31.520897 13514 leader_election.cc:302] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:31.521167 14823 raft_consensus.cc:2781] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:31.521296 14823 raft_consensus.cc:482] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:31.521376 14823 raft_consensus.cc:3037] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.521816 14897 raft_consensus.cc:504] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:31.522248 14897 leader_election.cc:288] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:31.522564 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ef68f6c2e4245dd86895c93ec270392" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:31.522711 13317 raft_consensus.cc:3037] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.522753 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ef68f6c2e4245dd86895c93ec270392" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:31.522910 13449 raft_consensus.cc:3037] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.523366 14823 raft_consensus.cc:504] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:31.523901 14823 leader_election.cc:288] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:31.524349 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdabe4a87ec4398b6e4508ccc0ae165" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:31.524520 13188 raft_consensus.cc:3037] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.524638 13317 raft_consensus.cc:2454] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:31.524983 13122 leader_election.cc:302] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:31.525243 14897 raft_consensus.cc:2781] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:31.525439 13449 raft_consensus.cc:2454] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:31.525532 14897 raft_consensus.cc:686] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:31.525583 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7bdabe4a87ec4398b6e4508ccc0ae165" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:31.525746 13448 raft_consensus.cc:3037] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.525868 14897 consensus_queue.cc:237] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:31.526830 13188 raft_consensus.cc:2454] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:31.528867 13448 raft_consensus.cc:2454] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:31.529253 13514 leader_election.cc:302] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:31.529650 14823 raft_consensus.cc:2781] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:31.529868 14823 raft_consensus.cc:686] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:31.530128 14823 consensus_queue.cc:237] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:31.531842 13036 catalog_manager.cc:5484] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:31.532909 13040 catalog_manager.cc:5484] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:31.853717 14823 consensus_queue.cc:1035] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:31.866520 13317 raft_consensus.cc:3037] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:31.869760 14845 consensus_queue.cc:1035] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:31.883437 14927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:31.883673 14927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:31.890280 14927 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240502 14:14:31.963691 14897 consensus_queue.cc:1035] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:31.981317 14824 consensus_queue.cc:1035] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:32.065531 14823 consensus_queue.cc:1035] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20240502 14:14:32.106459 13188 raft_consensus.cc:1261] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:32.107350 14845 consensus_queue.cc:1035] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 326.028 ms
time per row: 0.163014 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:14:33.222414 13038 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:33.230616 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 0e393a7736b14d5483973f9cb89058f8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.230824 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 0e393a7736b14d5483973f9cb89058f8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.230978 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e393a7736b14d5483973f9cb89058f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.231216 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e393a7736b14d5483973f9cb89058f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.234020 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 0e393a7736b14d5483973f9cb89058f8 (DEFAULT_TABLE table=rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.234149 13166 tablet_service.cc:1467] Processing CreateTablet for tablet b2905b7940fb45c49ef8b18943dae256 (DEFAULT_TABLE table=rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.234381 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e393a7736b14d5483973f9cb89058f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.234508 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2905b7940fb45c49ef8b18943dae256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.237764 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 80040c60b029481285c50171bcc5ce35 (DEFAULT_TABLE table=rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.238183 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80040c60b029481285c50171bcc5ce35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.239312 13428 tablet_service.cc:1467] Processing CreateTablet for tablet 80040c60b029481285c50171bcc5ce35 (DEFAULT_TABLE table=rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.239727 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80040c60b029481285c50171bcc5ce35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.241628 13427 tablet_service.cc:1467] Processing CreateTablet for tablet b2905b7940fb45c49ef8b18943dae256 (DEFAULT_TABLE table=rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.242019 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2905b7940fb45c49ef8b18943dae256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.243278 15001 tablet_bootstrap.cc:492] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:33.245079 15001 tablet_bootstrap.cc:654] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.245959 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 80040c60b029481285c50171bcc5ce35 (DEFAULT_TABLE table=rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.246416 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80040c60b029481285c50171bcc5ce35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.249650 13296 tablet_service.cc:1467] Processing CreateTablet for tablet b2905b7940fb45c49ef8b18943dae256 (DEFAULT_TABLE table=rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:33.250066 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2905b7940fb45c49ef8b18943dae256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:33.251252 15003 tablet_bootstrap.cc:492] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:33.253012 15003 tablet_bootstrap.cc:654] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.254529 15000 tablet_bootstrap.cc:492] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:33.256470 15000 tablet_bootstrap.cc:654] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.258172 15003 tablet_bootstrap.cc:492] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:33.258394 15000 tablet_bootstrap.cc:492] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:33.258513 15003 ts_tablet_manager.cc:1367] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240502 14:14:33.258612 15000 ts_tablet_manager.cc:1367] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:14:33.259301 15003 raft_consensus.cc:348] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:33.259541 15003 raft_consensus.cc:374] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.259660 15003 raft_consensus.cc:729] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.259583 15000 raft_consensus.cc:348] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:33.259990 15000 raft_consensus.cc:374] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.259938 15003 consensus_queue.cc:260] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:33.260314 15000 raft_consensus.cc:729] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.260509 15003 ts_tablet_manager.cc:1398] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:33.260845 15003 tablet_bootstrap.cc:492] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:33.260696 15000 consensus_queue.cc:260] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:33.261447 15000 ts_tablet_manager.cc:1398] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:33.263449 15003 tablet_bootstrap.cc:654] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.274372 15003 tablet_bootstrap.cc:492] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:33.274538 15003 ts_tablet_manager.cc:1367] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.008s
I20240502 14:14:33.275168 15003 raft_consensus.cc:348] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:33.275452 15003 raft_consensus.cc:374] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.275552 15003 raft_consensus.cc:729] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.275789 15003 consensus_queue.cc:260] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:33.276470 15003 ts_tablet_manager.cc:1398] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:33.278935 15000 tablet_bootstrap.cc:492] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:33.280534 15000 tablet_bootstrap.cc:654] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.282042 15007 tablet_bootstrap.cc:492] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:33.286145 15007 tablet_bootstrap.cc:654] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.288374 15001 tablet_bootstrap.cc:492] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:33.288538 15001 ts_tablet_manager.cc:1367] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.020s
I20240502 14:14:33.289193 15001 raft_consensus.cc:348] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:33.289455 15001 raft_consensus.cc:374] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.289584 15001 raft_consensus.cc:729] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.289826 15001 consensus_queue.cc:260] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:33.291066 15001 ts_tablet_manager.cc:1398] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:33.291326 15001 tablet_bootstrap.cc:492] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:33.293605 15001 tablet_bootstrap.cc:654] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.303540 15001 tablet_bootstrap.cc:492] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:33.303732 15001 ts_tablet_manager.cc:1367] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240502 14:14:33.304035 15000 tablet_bootstrap.cc:492] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:33.304217 15000 ts_tablet_manager.cc:1367] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.017s
I20240502 14:14:33.304273 15001 raft_consensus.cc:348] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:33.304440 15001 raft_consensus.cc:374] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.304461 15007 tablet_bootstrap.cc:492] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:33.304505 15001 raft_consensus.cc:729] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.304575 15007 ts_tablet_manager.cc:1367] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.010s
I20240502 14:14:33.304728 15001 consensus_queue.cc:260] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:33.304895 15000 raft_consensus.cc:348] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:33.305356 15000 raft_consensus.cc:374] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.305276 15007 raft_consensus.cc:348] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:33.305452 15000 raft_consensus.cc:729] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.305509 15007 raft_consensus.cc:374] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.305641 15007 raft_consensus.cc:729] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.305711 15000 consensus_queue.cc:260] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:33.305879 15007 consensus_queue.cc:260] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:33.306423 15007 ts_tablet_manager.cc:1398] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:33.307052 15000 ts_tablet_manager.cc:1398] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:33.307122 15001 ts_tablet_manager.cc:1398] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:33.307353 15000 tablet_bootstrap.cc:492] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:33.307420 15001 tablet_bootstrap.cc:492] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:33.308887 15000 tablet_bootstrap.cc:654] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.309026 15001 tablet_bootstrap.cc:654] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:33.314738 15001 tablet_bootstrap.cc:492] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:33.314841 15000 tablet_bootstrap.cc:492] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:33.314858 15001 ts_tablet_manager.cc:1367] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 14:14:33.314965 15000 ts_tablet_manager.cc:1367] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20240502 14:14:33.315344 15001 raft_consensus.cc:348] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:33.315475 15001 raft_consensus.cc:374] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.315433 15000 raft_consensus.cc:348] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:33.315555 15001 raft_consensus.cc:729] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.315582 15000 raft_consensus.cc:374] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:33.315655 15000 raft_consensus.cc:729] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:33.315739 15001 consensus_queue.cc:260] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:33.315842 15000 consensus_queue.cc:260] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:33.316176 15001 ts_tablet_manager.cc:1398] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:14:33.316334 15000 ts_tablet_manager.cc:1398] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:14:33.344903 14988 raft_consensus.cc:482] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:33.345028 14988 raft_consensus.cc:504] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:33.345594 14988 leader_election.cc:288] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:33.345927 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80040c60b029481285c50171bcc5ce35" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:33.345988 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80040c60b029481285c50171bcc5ce35" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:33.346138 13448 raft_consensus.cc:2454] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:33.346169 13318 raft_consensus.cc:2454] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:33.346522 13514 leader_election.cc:302] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:33.346755 14988 raft_consensus.cc:2781] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:33.346886 14988 raft_consensus.cc:482] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:33.346973 14988 raft_consensus.cc:3037] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.348702 14988 raft_consensus.cc:504] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:33.349126 14988 leader_election.cc:288] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:33.349449 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80040c60b029481285c50171bcc5ce35" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:33.349493 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80040c60b029481285c50171bcc5ce35" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:33.349635 13448 raft_consensus.cc:3037] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.349634 13318 raft_consensus.cc:3037] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.352049 13318 raft_consensus.cc:2454] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:33.352056 13448 raft_consensus.cc:2454] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:33.352383 13514 leader_election.cc:302] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:33.352639 14988 raft_consensus.cc:2781] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:33.353193 14988 raft_consensus.cc:686] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:33.353451 14988 consensus_queue.cc:237] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:33.355867 13036 catalog_manager.cc:5484] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:33.570133 14969 raft_consensus.cc:482] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:33.570294 14969 raft_consensus.cc:504] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:33.570861 14969 leader_election.cc:288] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:33.571270 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2905b7940fb45c49ef8b18943dae256" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:33.571308 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2905b7940fb45c49ef8b18943dae256" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:33.571504 13318 raft_consensus.cc:2454] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:33.571504 13448 raft_consensus.cc:2454] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:33.571877 13120 leader_election.cc:302] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:33.572158 14969 raft_consensus.cc:2781] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:33.572261 14969 raft_consensus.cc:482] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:33.572348 14969 raft_consensus.cc:3037] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.574178 14969 raft_consensus.cc:504] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:33.574673 14969 leader_election.cc:288] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:33.575119 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2905b7940fb45c49ef8b18943dae256" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:33.575307 13318 raft_consensus.cc:3037] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.575261 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2905b7940fb45c49ef8b18943dae256" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:33.575443 13448 raft_consensus.cc:3037] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.577368 13318 raft_consensus.cc:2454] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:33.577570 13448 raft_consensus.cc:2454] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:33.577698 13122 leader_election.cc:302] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:33.577981 14969 raft_consensus.cc:2781] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:33.578310 14969 raft_consensus.cc:686] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:33.578593 14969 consensus_queue.cc:237] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:33.581993 13036 catalog_manager.cc:5484] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:33.643746 14848 raft_consensus.cc:482] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:33.643886 14848 raft_consensus.cc:504] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:33.644495 14848 leader_election.cc:288] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:33.644807 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e393a7736b14d5483973f9cb89058f8" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:33.644925 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e393a7736b14d5483973f9cb89058f8" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:33.645043 13318 raft_consensus.cc:2454] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:33.645167 13579 raft_consensus.cc:2454] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:33.645478 13384 leader_election.cc:302] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:33.645733 14848 raft_consensus.cc:2781] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:33.645850 14848 raft_consensus.cc:482] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:33.645926 14848 raft_consensus.cc:3037] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.647699 14848 raft_consensus.cc:504] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:33.648180 14848 leader_election.cc:288] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:33.648519 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e393a7736b14d5483973f9cb89058f8" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:33.648613 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e393a7736b14d5483973f9cb89058f8" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:33.648677 13318 raft_consensus.cc:3037] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.648797 13579 raft_consensus.cc:3037] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:33.650583 13318 raft_consensus.cc:2454] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:33.650837 13579 raft_consensus.cc:2454] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:33.650949 13384 leader_election.cc:302] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:33.651224 14848 raft_consensus.cc:2781] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:33.651602 14848 raft_consensus.cc:686] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:33.651898 14848 consensus_queue.cc:237] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:33.654857 13036 catalog_manager.cc:5484] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:33.828343 15013 consensus_queue.cc:1035] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:33.838116 14994 consensus_queue.cc:1035] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:34.018754 15020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:34.019068 15020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:34.026151 15020 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:34.040385 14969 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:34.045758 15015 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
I20240502 14:14:34.103955 14848 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:34.141028 15017 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 274.844 ms
time per row: 0.137422 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:14:35.317044 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:35.325798 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 373143a499c0492991c0693caf245ff4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.326053 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 373143a499c0492991c0693caf245ff4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.326244 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373143a499c0492991c0693caf245ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.326568 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373143a499c0492991c0693caf245ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.326818 13168 tablet_service.cc:1467] Processing CreateTablet for tablet eaba1d442abf493bac8cce1c4ab9cf5c (DEFAULT_TABLE table=rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.327255 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaba1d442abf493bac8cce1c4ab9cf5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.329968 13559 tablet_service.cc:1467] Processing CreateTablet for tablet eaba1d442abf493bac8cce1c4ab9cf5c (DEFAULT_TABLE table=rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.330343 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaba1d442abf493bac8cce1c4ab9cf5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.333931 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 373143a499c0492991c0693caf245ff4 (DEFAULT_TABLE table=rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.334383 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 373143a499c0492991c0693caf245ff4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.334688 13428 tablet_service.cc:1467] Processing CreateTablet for tablet eaba1d442abf493bac8cce1c4ab9cf5c (DEFAULT_TABLE table=rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.335067 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaba1d442abf493bac8cce1c4ab9cf5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.335533 13561 tablet_service.cc:1467] Processing CreateTablet for tablet 525ac3ddd7a148e89bc3dc241e9854d0 (DEFAULT_TABLE table=rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.335923 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 525ac3ddd7a148e89bc3dc241e9854d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.337208 15094 tablet_bootstrap.cc:492] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:35.338383 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 525ac3ddd7a148e89bc3dc241e9854d0 (DEFAULT_TABLE table=rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.338738 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 525ac3ddd7a148e89bc3dc241e9854d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.342613 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 525ac3ddd7a148e89bc3dc241e9854d0 (DEFAULT_TABLE table=rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:35.343111 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 525ac3ddd7a148e89bc3dc241e9854d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:35.348431 15094 tablet_bootstrap.cc:654] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.349795 15097 tablet_bootstrap.cc:492] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:35.351505 15093 tablet_bootstrap.cc:492] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:35.351837 15097 tablet_bootstrap.cc:654] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.351999 15096 tablet_bootstrap.cc:492] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:35.352963 15093 tablet_bootstrap.cc:654] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.353672 15096 tablet_bootstrap.cc:654] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.355120 15093 tablet_bootstrap.cc:492] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:35.355316 15093 ts_tablet_manager.cc:1367] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:35.355333 15097 tablet_bootstrap.cc:492] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:35.355458 15097 ts_tablet_manager.cc:1367] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:14:35.355839 15096 tablet_bootstrap.cc:492] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:35.356012 15096 ts_tablet_manager.cc:1367] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:14:35.356266 15093 raft_consensus.cc:348] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:35.356269 15097 raft_consensus.cc:348] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:35.356433 15093 raft_consensus.cc:374] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.356452 15097 raft_consensus.cc:374] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.356532 15097 raft_consensus.cc:729] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.356532 15093 raft_consensus.cc:729] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.356768 15093 consensus_queue.cc:260] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:35.356783 15097 consensus_queue.cc:260] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:35.356823 15096 raft_consensus.cc:348] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:35.357028 15096 raft_consensus.cc:374] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.357177 15096 raft_consensus.cc:729] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.357364 15093 ts_tablet_manager.cc:1398] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:35.357474 15096 consensus_queue.cc:260] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:35.357682 15093 tablet_bootstrap.cc:492] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:35.357757 15097 ts_tablet_manager.cc:1398] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:35.358906 15096 ts_tablet_manager.cc:1398] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:35.359180 15096 tablet_bootstrap.cc:492] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:35.359709 15094 tablet_bootstrap.cc:492] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:35.359896 15094 ts_tablet_manager.cc:1367] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.002s
I20240502 14:14:35.360744 15094 raft_consensus.cc:348] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:35.360919 15096 tablet_bootstrap.cc:654] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.360993 15094 raft_consensus.cc:374] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.361130 15094 raft_consensus.cc:729] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.361374 15094 consensus_queue.cc:260] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:35.362054 15094 ts_tablet_manager.cc:1398] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:35.362320 15094 tablet_bootstrap.cc:492] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:35.363925 15094 tablet_bootstrap.cc:654] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.365159 15093 tablet_bootstrap.cc:654] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.368638 15094 tablet_bootstrap.cc:492] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:35.368783 15094 ts_tablet_manager.cc:1367] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20240502 14:14:35.369520 15094 raft_consensus.cc:348] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:35.369793 15094 raft_consensus.cc:374] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.369885 15094 raft_consensus.cc:729] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.370113 15094 consensus_queue.cc:260] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:35.370689 15094 ts_tablet_manager.cc:1398] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:35.370982 15094 tablet_bootstrap.cc:492] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:35.373526 15096 tablet_bootstrap.cc:492] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:35.373710 15096 ts_tablet_manager.cc:1367] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.005s
I20240502 14:14:35.373812 15094 tablet_bootstrap.cc:654] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.375162 15096 raft_consensus.cc:348] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:35.375365 15094 tablet_bootstrap.cc:492] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:35.375394 15096 raft_consensus.cc:374] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.375461 15096 raft_consensus.cc:729] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.375643 15094 ts_tablet_manager.cc:1367] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 14:14:35.375749 15096 consensus_queue.cc:260] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:35.376360 15096 ts_tablet_manager.cc:1398] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:35.376317 15094 raft_consensus.cc:348] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:35.376613 15094 raft_consensus.cc:374] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.376729 15094 raft_consensus.cc:729] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.376979 15094 consensus_queue.cc:260] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:35.377676 15094 ts_tablet_manager.cc:1398] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:35.381784 14848 raft_consensus.cc:482] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:35.381870 15093 tablet_bootstrap.cc:492] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:35.381929 14848 raft_consensus.cc:504] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:35.382025 15093 ts_tablet_manager.cc:1367] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.006s
I20240502 14:14:35.382665 14848 leader_election.cc:288] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:35.382702 15093 raft_consensus.cc:348] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:35.382900 15093 raft_consensus.cc:374] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.382977 15093 raft_consensus.cc:729] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.383185 15093 consensus_queue.cc:260] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:35.383460 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaba1d442abf493bac8cce1c4ab9cf5c" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:35.383649 13186 raft_consensus.cc:2454] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:35.383702 15093 ts_tablet_manager.cc:1398] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:35.383973 15093 tablet_bootstrap.cc:492] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:35.384079 13382 leader_election.cc:302] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:35.384311 14848 raft_consensus.cc:2781] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:35.384418 14848 raft_consensus.cc:482] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:35.384496 14848 raft_consensus.cc:3037] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.384588 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaba1d442abf493bac8cce1c4ab9cf5c" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:35.384750 13580 raft_consensus.cc:2454] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:35.385394 15093 tablet_bootstrap.cc:654] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:35.386493 14848 raft_consensus.cc:504] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:35.386925 14848 leader_election.cc:288] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:35.386961 15093 tablet_bootstrap.cc:492] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:35.387126 15093 ts_tablet_manager.cc:1367] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:35.387367 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaba1d442abf493bac8cce1c4ab9cf5c" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:35.387527 13580 raft_consensus.cc:3037] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.387830 15093 raft_consensus.cc:348] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:35.388106 15093 raft_consensus.cc:374] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:35.388195 15093 raft_consensus.cc:729] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:35.388813 15093 consensus_queue.cc:260] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:35.389446 15093 ts_tablet_manager.cc:1398] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240502 14:14:35.389883 13580 raft_consensus.cc:2454] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:35.390153 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaba1d442abf493bac8cce1c4ab9cf5c" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:35.390313 13186 raft_consensus.cc:3037] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.392540 13186 raft_consensus.cc:2454] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:35.392763 13382 leader_election.cc:302] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:35.393013 14848 raft_consensus.cc:2781] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:35.393174 14848 raft_consensus.cc:686] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:35.393431 14848 consensus_queue.cc:237] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:35.395830 13036 catalog_manager.cc:5484] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:35.431408 15068 raft_consensus.cc:482] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:35.431543 15068 raft_consensus.cc:504] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:35.432018 15068 leader_election.cc:288] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:35.432371 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ac3ddd7a148e89bc3dc241e9854d0" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:35.432433 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ac3ddd7a148e89bc3dc241e9854d0" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:35.432544 13186 raft_consensus.cc:2454] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:35.432642 13580 raft_consensus.cc:2454] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:35.432955 13251 leader_election.cc:302] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:35.433163 15068 raft_consensus.cc:2781] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:35.433298 15068 raft_consensus.cc:482] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:35.433370 15068 raft_consensus.cc:3037] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.434927 15068 raft_consensus.cc:504] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:35.435328 15068 leader_election.cc:288] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:35.435712 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ac3ddd7a148e89bc3dc241e9854d0" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:35.435724 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "525ac3ddd7a148e89bc3dc241e9854d0" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:35.435869 13186 raft_consensus.cc:3037] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.435884 13580 raft_consensus.cc:3037] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.438414 13186 raft_consensus.cc:2454] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:35.438421 13580 raft_consensus.cc:2454] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:35.438714 13251 leader_election.cc:302] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:35.439079 15068 raft_consensus.cc:2781] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:35.439371 15068 raft_consensus.cc:686] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:35.439638 15068 consensus_queue.cc:237] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:35.442422 13036 catalog_manager.cc:5484] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:35.519717 14969 raft_consensus.cc:482] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:35.519945 14969 raft_consensus.cc:504] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:35.520771 14969 leader_election.cc:288] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:35.521296 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373143a499c0492991c0693caf245ff4" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:35.521340 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373143a499c0492991c0693caf245ff4" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:35.521526 13579 raft_consensus.cc:2454] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:35.521533 13319 raft_consensus.cc:2454] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:35.521908 13120 leader_election.cc:302] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:35.522224 14969 raft_consensus.cc:2781] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:35.522311 14969 raft_consensus.cc:482] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:35.522383 14969 raft_consensus.cc:3037] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.524066 14969 raft_consensus.cc:504] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:35.524546 14969 leader_election.cc:288] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:35.524892 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373143a499c0492991c0693caf245ff4" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:35.524922 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "373143a499c0492991c0693caf245ff4" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:35.525048 13579 raft_consensus.cc:3037] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.525097 13319 raft_consensus.cc:3037] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:35.526860 13579 raft_consensus.cc:2454] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:35.526945 13319 raft_consensus.cc:2454] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:35.527218 13120 leader_election.cc:302] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:35.527452 14969 raft_consensus.cc:2781] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:35.527587 14969 raft_consensus.cc:686] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:35.527866 14969 consensus_queue.cc:237] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:35.530418 13036 catalog_manager.cc:5484] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:35.839116 15108 consensus_queue.cc:1035] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:35.845597 15108 consensus_queue.cc:1035] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:35.857419 15111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:35.857825 15111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:35.870157 15111 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_10'
I20240502 14:14:36.009335 13319 raft_consensus.cc:1261] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:36.010020 14969 consensus_queue.cc:1035] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:36.013556 14969 consensus_queue.cc:1035] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:36.033171 14848 consensus_queue.cc:1035] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:36.034040 14848 consensus_queue.cc:1035] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 289.744 ms
time per row: 0.144872 ms
I20240502 14:14:36.302505 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:36.311096 13428 tablet_service.cc:1467] Processing CreateTablet for tablet abd272455c064ba48726d933d53c83ed (DEFAULT_TABLE table=rebalance_test_table_11 [id=b7310fe1c01b4fe7b29328321c07e72d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.311352 13561 tablet_service.cc:1467] Processing CreateTablet for tablet 2e51feadc94c48e9bcecffecacbda4b9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=b7310fe1c01b4fe7b29328321c07e72d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.311548 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd272455c064ba48726d933d53c83ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.311744 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e51feadc94c48e9bcecffecacbda4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.312088 13168 tablet_service.cc:1467] Processing CreateTablet for tablet abd272455c064ba48726d933d53c83ed (DEFAULT_TABLE table=rebalance_test_table_11 [id=b7310fe1c01b4fe7b29328321c07e72d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.312276 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 652e49046212458280095c016a342f47 (DEFAULT_TABLE table=rebalance_test_table_11 [id=b7310fe1c01b4fe7b29328321c07e72d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.312436 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd272455c064ba48726d933d53c83ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.312613 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652e49046212458280095c016a342f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.317391 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 2e51feadc94c48e9bcecffecacbda4b9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=b7310fe1c01b4fe7b29328321c07e72d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.317719 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e51feadc94c48e9bcecffecacbda4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.318270 15181 tablet_bootstrap.cc:492] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:36.318300 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 2e51feadc94c48e9bcecffecacbda4b9 (DEFAULT_TABLE table=rebalance_test_table_11 [id=b7310fe1c01b4fe7b29328321c07e72d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.318645 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e51feadc94c48e9bcecffecacbda4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.318743 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 652e49046212458280095c016a342f47 (DEFAULT_TABLE table=rebalance_test_table_11 [id=b7310fe1c01b4fe7b29328321c07e72d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.319007 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652e49046212458280095c016a342f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.320345 15181 tablet_bootstrap.cc:654] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.321327 13297 tablet_service.cc:1467] Processing CreateTablet for tablet abd272455c064ba48726d933d53c83ed (DEFAULT_TABLE table=rebalance_test_table_11 [id=b7310fe1c01b4fe7b29328321c07e72d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.321707 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd272455c064ba48726d933d53c83ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.321939 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 652e49046212458280095c016a342f47 (DEFAULT_TABLE table=rebalance_test_table_11 [id=b7310fe1c01b4fe7b29328321c07e72d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:36.322268 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652e49046212458280095c016a342f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:36.329419 15180 tablet_bootstrap.cc:492] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:36.331359 15180 tablet_bootstrap.cc:654] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.334023 15179 tablet_bootstrap.cc:492] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:36.335721 15179 tablet_bootstrap.cc:654] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.337929 15182 tablet_bootstrap.cc:492] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:36.339311 15180 tablet_bootstrap.cc:492] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:36.339465 15180 ts_tablet_manager.cc:1367] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.005s
I20240502 14:14:36.339646 15181 tablet_bootstrap.cc:492] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:36.339804 15181 ts_tablet_manager.cc:1367] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20240502 14:14:36.340267 15180 raft_consensus.cc:348] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:36.340476 15180 raft_consensus.cc:374] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.340564 15182 tablet_bootstrap.cc:654] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.340629 15180 raft_consensus.cc:729] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.340833 15181 raft_consensus.cc:348] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.340903 15180 consensus_queue.cc:260] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:36.341117 15181 raft_consensus.cc:374] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.341264 15181 raft_consensus.cc:729] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.341544 15180 ts_tablet_manager.cc:1398] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:36.341533 15181 consensus_queue.cc:260] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.342115 15181 ts_tablet_manager.cc:1398] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:36.342845 15180 tablet_bootstrap.cc:492] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:36.344136 15180 tablet_bootstrap.cc:654] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.345367 15181 tablet_bootstrap.cc:492] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:36.346294 15179 tablet_bootstrap.cc:492] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:36.346511 15179 ts_tablet_manager.cc:1367] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240502 14:14:36.349769 15179 raft_consensus.cc:348] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.350096 15179 raft_consensus.cc:374] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.350241 15179 raft_consensus.cc:729] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.350597 15179 consensus_queue.cc:260] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.351089 15181 tablet_bootstrap.cc:654] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.352358 15179 ts_tablet_manager.cc:1398] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:14:36.352715 15179 tablet_bootstrap.cc:492] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:36.355657 15179 tablet_bootstrap.cc:654] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.358469 15181 tablet_bootstrap.cc:492] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:36.358656 15181 ts_tablet_manager.cc:1367] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 14:14:36.359329 15181 raft_consensus.cc:348] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:36.359557 15181 raft_consensus.cc:374] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.359663 15181 raft_consensus.cc:729] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.359756 15182 tablet_bootstrap.cc:492] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:36.359961 15180 tablet_bootstrap.cc:492] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:36.360014 15182 ts_tablet_manager.cc:1367] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.014s
I20240502 14:14:36.359932 15181 consensus_queue.cc:260] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:36.360093 15180 ts_tablet_manager.cc:1367] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.014s
I20240502 14:14:36.360518 15181 ts_tablet_manager.cc:1398] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:36.360749 15180 raft_consensus.cc:348] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.360889 15180 raft_consensus.cc:374] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.360930 15180 raft_consensus.cc:729] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.360929 15182 raft_consensus.cc:348] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.361129 15180 consensus_queue.cc:260] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.361188 15182 raft_consensus.cc:374] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.361301 15182 raft_consensus.cc:729] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.361537 15180 ts_tablet_manager.cc:1398] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:14:36.361572 15182 consensus_queue.cc:260] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.362223 15182 ts_tablet_manager.cc:1398] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:36.364009 15179 tablet_bootstrap.cc:492] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:36.364270 15179 ts_tablet_manager.cc:1367] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240502 14:14:36.364940 15182 tablet_bootstrap.cc:492] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:36.364972 15179 raft_consensus.cc:348] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.365206 15179 raft_consensus.cc:374] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.365337 15179 raft_consensus.cc:729] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.365624 15179 consensus_queue.cc:260] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.366425 15182 tablet_bootstrap.cc:654] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.366941 15179 ts_tablet_manager.cc:1398] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:14:36.367290 15179 tablet_bootstrap.cc:492] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:36.369387 15179 tablet_bootstrap.cc:654] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:36.370937 15179 tablet_bootstrap.cc:492] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:36.371050 15179 ts_tablet_manager.cc:1367] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:14:36.371600 15179 raft_consensus.cc:348] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:36.371753 15179 raft_consensus.cc:374] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.371837 15179 raft_consensus.cc:729] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.372009 15179 consensus_queue.cc:260] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:36.373010 15179 ts_tablet_manager.cc:1398] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:36.374891 15182 tablet_bootstrap.cc:492] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:36.375229 15182 ts_tablet_manager.cc:1367] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 14:14:36.375813 15182 raft_consensus.cc:348] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.375963 15182 raft_consensus.cc:374] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:36.376036 15182 raft_consensus.cc:729] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:36.376477 15182 consensus_queue.cc:260] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.377940 15182 ts_tablet_manager.cc:1398] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:36.414686 15112 raft_consensus.cc:482] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:36.414811 15112 raft_consensus.cc:504] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.415729 15112 leader_election.cc:288] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:36.416009 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652e49046212458280095c016a342f47" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:36.416267 13449 raft_consensus.cc:2454] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:36.416628 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652e49046212458280095c016a342f47" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:36.416735 13514 leader_election.cc:302] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:36.416774 15108 raft_consensus.cc:482] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:36.416968 15112 raft_consensus.cc:2781] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:36.416918 15108 raft_consensus.cc:504] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.417110 15112 raft_consensus.cc:482] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:36.417196 15112 raft_consensus.cc:3037] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.417801 15108 leader_election.cc:288] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:36.417995 13318 raft_consensus.cc:2454] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:36.418120 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652e49046212458280095c016a342f47" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:36.418323 13449 raft_consensus.cc:2454] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:36.418406 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652e49046212458280095c016a342f47" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:36.418875 13251 leader_election.cc:302] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:36.419137 15108 raft_consensus.cc:2781] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:36.419246 15108 raft_consensus.cc:482] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:36.419332 15108 raft_consensus.cc:3037] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.419745 15112 raft_consensus.cc:504] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.420164 13579 raft_consensus.cc:2379] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate cd67493f6f0142f896010f2b8a39887d in current term 1: Already voted for candidate 7eb4ecb2379b44c0954f2658c4a04350 in this term.
I20240502 14:14:36.420841 15112 leader_election.cc:288] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:36.420881 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652e49046212458280095c016a342f47" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:36.421759 15108 raft_consensus.cc:504] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.422040 13318 raft_consensus.cc:2379] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7eb4ecb2379b44c0954f2658c4a04350 in current term 1: Already voted for candidate cd67493f6f0142f896010f2b8a39887d in this term.
I20240502 14:14:36.422214 15108 leader_election.cc:288] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:36.422439 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652e49046212458280095c016a342f47" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:36.422502 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652e49046212458280095c016a342f47" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:36.422631 13449 raft_consensus.cc:3037] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.422613 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "652e49046212458280095c016a342f47" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:36.422804 13579 raft_consensus.cc:2379] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cd67493f6f0142f896010f2b8a39887d in current term 1: Already voted for candidate 7eb4ecb2379b44c0954f2658c4a04350 in this term.
I20240502 14:14:36.423182 13251 leader_election.cc:302] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d; no voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:14:36.423413 15108 raft_consensus.cc:2726] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:14:36.424410 13449 raft_consensus.cc:2454] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:36.424729 13514 leader_election.cc:302] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters: cd67493f6f0142f896010f2b8a39887d
I20240502 14:14:36.424986 15112 raft_consensus.cc:2781] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:36.425097 15112 raft_consensus.cc:686] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:36.425336 15112 consensus_queue.cc:237] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.428052 13036 catalog_manager.cc:5484] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:36.430682 15161 raft_consensus.cc:482] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:36.430860 15161 raft_consensus.cc:504] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:36.431597 15161 leader_election.cc:288] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:36.431874 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e51feadc94c48e9bcecffecacbda4b9" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:36.431954 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e51feadc94c48e9bcecffecacbda4b9" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:36.432067 13579 raft_consensus.cc:2454] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:36.432114 13449 raft_consensus.cc:2454] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:36.432397 13120 leader_election.cc:302] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:36.432596 15161 raft_consensus.cc:2781] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:36.432680 15161 raft_consensus.cc:482] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:36.432751 15161 raft_consensus.cc:3037] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.434303 15161 raft_consensus.cc:504] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:36.434726 15161 leader_election.cc:288] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:36.435068 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e51feadc94c48e9bcecffecacbda4b9" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:36.435106 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e51feadc94c48e9bcecffecacbda4b9" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:36.435207 13579 raft_consensus.cc:3037] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.435256 13449 raft_consensus.cc:3037] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.437086 13579 raft_consensus.cc:2454] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:36.437431 13120 leader_election.cc:302] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:36.437701 15161 raft_consensus.cc:2781] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:36.437848 15161 raft_consensus.cc:686] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:36.438089 15161 consensus_queue.cc:237] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:36.438935 13449 raft_consensus.cc:2454] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:36.440950 13036 catalog_manager.cc:5484] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:36.469074 15156 raft_consensus.cc:482] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:36.469185 15156 raft_consensus.cc:504] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.469480 15161 raft_consensus.cc:482] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:36.469621 15161 raft_consensus.cc:504] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.469938 15156 leader_election.cc:288] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:36.470196 15161 leader_election.cc:288] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:36.470248 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd272455c064ba48726d933d53c83ed" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:36.470317 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd272455c064ba48726d933d53c83ed" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:36.470435 13187 raft_consensus.cc:2454] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:36.470523 13318 raft_consensus.cc:2454] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:36.470708 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd272455c064ba48726d933d53c83ed" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:36.470785 13382 leader_election.cc:302] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:36.470901 13319 raft_consensus.cc:2454] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:36.471009 15156 raft_consensus.cc:2781] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:36.471101 15156 raft_consensus.cc:482] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:36.471190 15156 raft_consensus.cc:3037] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.471184 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd272455c064ba48726d933d53c83ed" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:36.471261 13122 leader_election.cc:302] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:36.471482 15161 raft_consensus.cc:2781] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:36.471549 15161 raft_consensus.cc:482] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:36.471613 15161 raft_consensus.cc:3037] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.472797 15156 raft_consensus.cc:504] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.473121 13449 raft_consensus.cc:2379] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20a967bad5504b1e9c6fb720ecdd1625 in current term 1: Already voted for candidate 88b77f6edc1c4a72b548bf47c05704f3 in this term.
I20240502 14:14:36.473084 15161 raft_consensus.cc:504] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.473356 15156 leader_election.cc:288] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:36.473636 15161 leader_election.cc:288] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:36.473631 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd272455c064ba48726d933d53c83ed" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:36.473875 13187 raft_consensus.cc:2379] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 88b77f6edc1c4a72b548bf47c05704f3 in current term 1: Already voted for candidate 20a967bad5504b1e9c6fb720ecdd1625 in this term.
I20240502 14:14:36.473874 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd272455c064ba48726d933d53c83ed" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:36.474084 13448 raft_consensus.cc:2379] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 20a967bad5504b1e9c6fb720ecdd1625 in current term 1: Already voted for candidate 88b77f6edc1c4a72b548bf47c05704f3 in this term.
I20240502 14:14:36.474130 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd272455c064ba48726d933d53c83ed" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:36.474275 13319 raft_consensus.cc:3037] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:36.474702 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abd272455c064ba48726d933d53c83ed" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:36.475165 13122 leader_election.cc:302] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d
I20240502 14:14:36.475356 15161 raft_consensus.cc:2726] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:14:36.476109 13319 raft_consensus.cc:2454] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:36.476378 13384 leader_election.cc:302] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters: 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:14:36.476608 15156 raft_consensus.cc:2781] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:36.476708 15156 raft_consensus.cc:686] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:36.476956 15156 consensus_queue.cc:237] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:36.479480 13036 catalog_manager.cc:5484] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:36.792554 15195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:36.792798 15195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:36.799575 15195 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240502 14:14:36.869938 15156 consensus_queue.cc:1035] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:36.878000 15161 consensus_queue.cc:1035] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:36.914291 14848 consensus_queue.cc:1035] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:36.927248 14969 consensus_queue.cc:1035] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:36.941313 15112 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:36.971194 15215 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.6.196.254:44303, user_credentials={real_user=slave}} blocked reactor thread for 62541.1us
I20240502 14:14:37.005746 15237 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 322.911 ms
time per row: 0.161455 ms
-----------------------------------------------------
Suppressions used:
count bytes template
30 7844 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:14:38.176924 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:38.185658 13296 tablet_service.cc:1467] Processing CreateTablet for tablet d7b9a8e040f341ca8e25727310994df0 (DEFAULT_TABLE table=rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.185724 13427 tablet_service.cc:1467] Processing CreateTablet for tablet d7b9a8e040f341ca8e25727310994df0 (DEFAULT_TABLE table=rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.185904 13166 tablet_service.cc:1467] Processing CreateTablet for tablet d7b9a8e040f341ca8e25727310994df0 (DEFAULT_TABLE table=rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.186142 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7b9a8e040f341ca8e25727310994df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.186179 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7b9a8e040f341ca8e25727310994df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.186303 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7b9a8e040f341ca8e25727310994df0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.187551 13168 tablet_service.cc:1467] Processing CreateTablet for tablet f966f36415994f9eaa8046b407135761 (DEFAULT_TABLE table=rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.187947 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f966f36415994f9eaa8046b407135761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.189934 13167 tablet_service.cc:1467] Processing CreateTablet for tablet dcf9e4934b4d4400884eee33f42b29f5 (DEFAULT_TABLE table=rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.189939 13561 tablet_service.cc:1467] Processing CreateTablet for tablet dcf9e4934b4d4400884eee33f42b29f5 (DEFAULT_TABLE table=rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.190272 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf9e4934b4d4400884eee33f42b29f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.190280 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf9e4934b4d4400884eee33f42b29f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.193395 13428 tablet_service.cc:1467] Processing CreateTablet for tablet dcf9e4934b4d4400884eee33f42b29f5 (DEFAULT_TABLE table=rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.193743 13560 tablet_service.cc:1467] Processing CreateTablet for tablet f966f36415994f9eaa8046b407135761 (DEFAULT_TABLE table=rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.193861 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf9e4934b4d4400884eee33f42b29f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.194077 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f966f36415994f9eaa8046b407135761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.202945 15272 tablet_bootstrap.cc:492] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:38.203259 13297 tablet_service.cc:1467] Processing CreateTablet for tablet f966f36415994f9eaa8046b407135761 (DEFAULT_TABLE table=rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:38.203641 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f966f36415994f9eaa8046b407135761. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:38.206821 15272 tablet_bootstrap.cc:654] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.215338 15275 tablet_bootstrap.cc:492] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:38.216441 15272 tablet_bootstrap.cc:492] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:38.217027 15272 ts_tablet_manager.cc:1367] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20240502 14:14:38.217500 15275 tablet_bootstrap.cc:654] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.218366 15272 raft_consensus.cc:348] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:38.219086 15271 tablet_bootstrap.cc:492] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:38.219331 15272 raft_consensus.cc:374] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.219563 15272 raft_consensus.cc:729] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.220391 15272 consensus_queue.cc:260] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:38.221205 15271 tablet_bootstrap.cc:654] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.221808 15272 ts_tablet_manager.cc:1398] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:14:38.222079 15272 tablet_bootstrap.cc:492] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:38.223757 15272 tablet_bootstrap.cc:654] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.227581 15275 tablet_bootstrap.cc:492] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:38.227886 15270 tablet_bootstrap.cc:492] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:38.228127 15275 ts_tablet_manager.cc:1367] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20240502 14:14:38.228754 15275 raft_consensus.cc:348] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:38.228977 15275 raft_consensus.cc:374] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.229120 15275 raft_consensus.cc:729] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.229343 15275 consensus_queue.cc:260] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:38.229581 15270 tablet_bootstrap.cc:654] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.230026 15275 ts_tablet_manager.cc:1398] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:38.230193 15272 tablet_bootstrap.cc:492] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:38.230341 15275 tablet_bootstrap.cc:492] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:38.230429 15272 ts_tablet_manager.cc:1367] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20240502 14:14:38.231159 15272 raft_consensus.cc:348] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:38.231407 15272 raft_consensus.cc:374] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.231539 15272 raft_consensus.cc:729] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.231813 15272 consensus_queue.cc:260] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:38.232380 15272 ts_tablet_manager.cc:1398] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:38.232594 15271 tablet_bootstrap.cc:492] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:38.232717 15271 ts_tablet_manager.cc:1367] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20240502 14:14:38.233459 15272 tablet_bootstrap.cc:492] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:38.233408 15271 raft_consensus.cc:348] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:38.233604 15271 raft_consensus.cc:374] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.233692 15271 raft_consensus.cc:729] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.233893 15271 consensus_queue.cc:260] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:38.234396 15271 ts_tablet_manager.cc:1398] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:38.231751 15275 tablet_bootstrap.cc:654] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.235138 15272 tablet_bootstrap.cc:654] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.236310 15271 tablet_bootstrap.cc:492] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:38.239356 15271 tablet_bootstrap.cc:654] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.241480 15275 tablet_bootstrap.cc:492] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:38.241628 15275 ts_tablet_manager.cc:1367] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20240502 14:14:38.242450 15275 raft_consensus.cc:348] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:38.242697 15275 raft_consensus.cc:374] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.242771 15275 raft_consensus.cc:729] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.243010 15275 consensus_queue.cc:260] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:38.243583 15275 ts_tablet_manager.cc:1398] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:38.243808 15272 tablet_bootstrap.cc:492] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:38.243988 15272 ts_tablet_manager.cc:1367] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20240502 14:14:38.244720 15271 tablet_bootstrap.cc:492] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:38.244681 15272 raft_consensus.cc:348] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:38.244874 15271 ts_tablet_manager.cc:1367] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240502 14:14:38.244989 15272 raft_consensus.cc:374] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.245126 15272 raft_consensus.cc:729] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.245389 15272 consensus_queue.cc:260] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:38.245635 15271 raft_consensus.cc:348] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:38.245857 15271 raft_consensus.cc:374] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.245931 15271 raft_consensus.cc:729] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.246223 15271 consensus_queue.cc:260] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:38.246824 15272 ts_tablet_manager.cc:1398] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:38.247081 15271 ts_tablet_manager.cc:1398] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:38.248031 15270 tablet_bootstrap.cc:492] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:38.248178 15270 ts_tablet_manager.cc:1367] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.010s
I20240502 14:14:38.248867 15270 raft_consensus.cc:348] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:38.249015 15270 raft_consensus.cc:374] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.249908 15270 raft_consensus.cc:729] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.250119 15270 consensus_queue.cc:260] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:38.250715 15270 ts_tablet_manager.cc:1398] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:38.251036 15270 tablet_bootstrap.cc:492] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:38.252645 15270 tablet_bootstrap.cc:654] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:38.254247 15270 tablet_bootstrap.cc:492] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:38.254400 15270 ts_tablet_manager.cc:1367] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240502 14:14:38.254972 15270 raft_consensus.cc:348] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:38.255192 15270 raft_consensus.cc:374] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:38.255267 15270 raft_consensus.cc:729] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:38.255496 15270 consensus_queue.cc:260] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:38.255975 15270 ts_tablet_manager.cc:1398] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:14:38.283380 14848 raft_consensus.cc:482] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:38.283505 14848 raft_consensus.cc:504] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:38.284037 14848 leader_election.cc:288] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:38.284341 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b9a8e040f341ca8e25727310994df0" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:38.284442 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b9a8e040f341ca8e25727310994df0" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:38.284495 13319 raft_consensus.cc:2454] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:38.284647 13187 raft_consensus.cc:2454] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:38.284842 13384 leader_election.cc:302] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:38.285270 14848 raft_consensus.cc:2781] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:38.285378 14848 raft_consensus.cc:482] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:38.285462 14848 raft_consensus.cc:3037] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.286734 15286 raft_consensus.cc:482] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:38.286877 15286 raft_consensus.cc:504] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:38.287533 15286 leader_election.cc:288] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:38.287564 14848 raft_consensus.cc:504] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:38.287961 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf9e4934b4d4400884eee33f42b29f5" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:38.288090 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf9e4934b4d4400884eee33f42b29f5" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:38.288154 13187 raft_consensus.cc:2454] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:38.288339 13581 raft_consensus.cc:2454] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:38.288606 13382 leader_election.cc:302] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:38.288784 14848 leader_election.cc:288] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:38.288709 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b9a8e040f341ca8e25727310994df0" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:38.288901 13319 raft_consensus.cc:3037] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.288941 14848 raft_consensus.cc:2781] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:38.289012 14848 raft_consensus.cc:482] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:38.289098 14848 raft_consensus.cc:3037] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.291452 13319 raft_consensus.cc:2454] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:38.291440 14848 raft_consensus.cc:504] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:38.291764 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7b9a8e040f341ca8e25727310994df0" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:38.291814 13384 leader_election.cc:302] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:38.291939 13187 raft_consensus.cc:3037] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.292279 15286 raft_consensus.cc:2781] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:38.292268 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf9e4934b4d4400884eee33f42b29f5" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:38.292434 13186 raft_consensus.cc:3037] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.293718 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcf9e4934b4d4400884eee33f42b29f5" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:38.293963 13581 raft_consensus.cc:3037] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.294498 13187 raft_consensus.cc:2454] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:38.294792 13186 raft_consensus.cc:2454] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:38.295158 13382 leader_election.cc:302] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:38.295744 15288 raft_consensus.cc:2781] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:38.295882 15288 raft_consensus.cc:686] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:38.296108 15288 consensus_queue.cc:237] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:38.297693 13581 raft_consensus.cc:2454] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:38.297994 14848 leader_election.cc:288] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:38.298569 15286 raft_consensus.cc:686] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:38.298784 15286 consensus_queue.cc:237] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:38.301532 13036 catalog_manager.cc:5484] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:38.302402 15265 raft_consensus.cc:482] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:38.302536 15265 raft_consensus.cc:504] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:38.303082 15265 leader_election.cc:288] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:38.303460 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f966f36415994f9eaa8046b407135761" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:38.303634 13581 raft_consensus.cc:2454] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:38.303994 13120 leader_election.cc:302] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:38.303994 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f966f36415994f9eaa8046b407135761" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:38.304194 13319 raft_consensus.cc:2454] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:38.304247 15265 raft_consensus.cc:2781] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:38.304334 15265 raft_consensus.cc:482] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:38.304423 15265 raft_consensus.cc:3037] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.306674 15265 raft_consensus.cc:504] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:38.307253 13036 catalog_manager.cc:5484] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:38.307554 15265 leader_election.cc:288] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:38.307766 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f966f36415994f9eaa8046b407135761" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:38.307793 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f966f36415994f9eaa8046b407135761" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:38.307950 13581 raft_consensus.cc:3037] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.307965 13319 raft_consensus.cc:3037] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:38.310408 13319 raft_consensus.cc:2454] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:38.310420 13581 raft_consensus.cc:2454] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:38.310740 13120 leader_election.cc:302] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:38.311103 15265 raft_consensus.cc:2781] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:38.311216 15265 raft_consensus.cc:686] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:38.311445 15265 consensus_queue.cc:237] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:38.314293 13036 catalog_manager.cc:5484] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:38.648622 15292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:38.648880 15292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:38.655769 15292 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
I20240502 14:14:38.760641 15286 consensus_queue.cc:1035] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:38.767351 14848 consensus_queue.cc:1035] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:38.781289 15286 consensus_queue.cc:1035] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:38.795662 15287 consensus_queue.cc:1035] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:38.801522 15265 consensus_queue.cc:1035] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:38.844842 14969 consensus_queue.cc:1035] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 264.087 ms
time per row: 0.132043 ms
I20240502 14:14:39.058365 13036 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:39.068013 13296 tablet_service.cc:1467] Processing CreateTablet for tablet d5975e02205a43d5b93ed8ab51f8a0f8 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c0cb4710334148e4b976992ee52cd47c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.068292 13427 tablet_service.cc:1467] Processing CreateTablet for tablet d5975e02205a43d5b93ed8ab51f8a0f8 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c0cb4710334148e4b976992ee52cd47c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.068488 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5975e02205a43d5b93ed8ab51f8a0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.068688 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5975e02205a43d5b93ed8ab51f8a0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.073675 13428 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec7ae12c17247e9b8c03659435bd4e2 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c0cb4710334148e4b976992ee52cd47c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.073848 13560 tablet_service.cc:1467] Processing CreateTablet for tablet d5975e02205a43d5b93ed8ab51f8a0f8 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c0cb4710334148e4b976992ee52cd47c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.073927 13561 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec7ae12c17247e9b8c03659435bd4e2 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c0cb4710334148e4b976992ee52cd47c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.074033 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec7ae12c17247e9b8c03659435bd4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.074327 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5975e02205a43d5b93ed8ab51f8a0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.074517 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec7ae12c17247e9b8c03659435bd4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.077759 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 3d39f694c8554638a3b10ebf7cd01350 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c0cb4710334148e4b976992ee52cd47c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.078336 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d39f694c8554638a3b10ebf7cd01350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.081683 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 3d39f694c8554638a3b10ebf7cd01350 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c0cb4710334148e4b976992ee52cd47c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.082154 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d39f694c8554638a3b10ebf7cd01350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.087311 15360 tablet_bootstrap.cc:492] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:39.089419 15360 tablet_bootstrap.cc:654] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.091621 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec7ae12c17247e9b8c03659435bd4e2 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c0cb4710334148e4b976992ee52cd47c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.092031 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec7ae12c17247e9b8c03659435bd4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.093021 15359 tablet_bootstrap.cc:492] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:39.095259 15359 tablet_bootstrap.cc:654] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.097893 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 3d39f694c8554638a3b10ebf7cd01350 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c0cb4710334148e4b976992ee52cd47c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:39.098573 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d39f694c8554638a3b10ebf7cd01350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:39.099627 15360 tablet_bootstrap.cc:492] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:39.099761 15360 ts_tablet_manager.cc:1367] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20240502 14:14:39.100438 15360 raft_consensus.cc:348] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.100574 15360 raft_consensus.cc:374] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.100628 15360 raft_consensus.cc:729] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.100812 15360 consensus_queue.cc:260] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.101349 15360 ts_tablet_manager.cc:1398] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:14:39.101759 15360 tablet_bootstrap.cc:492] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:39.102233 15359 tablet_bootstrap.cc:492] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:39.102401 15359 ts_tablet_manager.cc:1367] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20240502 14:14:39.103370 15360 tablet_bootstrap.cc:654] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.104913 15359 raft_consensus.cc:348] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.105276 15359 raft_consensus.cc:374] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.105443 15359 raft_consensus.cc:729] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.105830 15359 consensus_queue.cc:260] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.108295 15359 ts_tablet_manager.cc:1398] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:14:39.108707 15359 tablet_bootstrap.cc:492] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:39.110648 15359 tablet_bootstrap.cc:654] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.113776 15366 tablet_bootstrap.cc:492] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:39.114370 15364 tablet_bootstrap.cc:492] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:39.116106 15366 tablet_bootstrap.cc:654] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.116102 15364 tablet_bootstrap.cc:654] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.119223 15359 tablet_bootstrap.cc:492] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:39.119382 15359 ts_tablet_manager.cc:1367] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240502 14:14:39.120158 15359 raft_consensus.cc:348] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:39.120365 15359 raft_consensus.cc:374] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.120430 15359 raft_consensus.cc:729] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.120654 15359 consensus_queue.cc:260] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:39.121677 15359 ts_tablet_manager.cc:1398] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:39.123322 15359 tablet_bootstrap.cc:492] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:39.125823 15359 tablet_bootstrap.cc:654] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.130755 15364 tablet_bootstrap.cc:492] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:39.130937 15364 ts_tablet_manager.cc:1367] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.005s
I20240502 14:14:39.131116 15359 tablet_bootstrap.cc:492] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:39.131240 15359 ts_tablet_manager.cc:1367] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240502 14:14:39.131830 15360 tablet_bootstrap.cc:492] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:39.131778 15364 raft_consensus.cc:348] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:39.131875 15359 raft_consensus.cc:348] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.132000 15360 ts_tablet_manager.cc:1367] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.030s user 0.011s sys 0.000s
I20240502 14:14:39.132045 15359 raft_consensus.cc:374] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.132135 15359 raft_consensus.cc:729] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.132138 15364 raft_consensus.cc:374] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.132287 15364 raft_consensus.cc:729] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.132570 15364 consensus_queue.cc:260] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:39.132789 15360 raft_consensus.cc:348] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.133000 15360 raft_consensus.cc:374] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.133082 15360 raft_consensus.cc:729] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.133342 15360 consensus_queue.cc:260] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.133694 15359 consensus_queue.cc:260] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.133973 15360 ts_tablet_manager.cc:1398] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:39.134863 15364 ts_tablet_manager.cc:1398] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:14:39.135147 15364 tablet_bootstrap.cc:492] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:39.136164 15359 ts_tablet_manager.cc:1398] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:14:39.137446 15366 tablet_bootstrap.cc:492] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:39.137583 15366 ts_tablet_manager.cc:1367] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.009s
I20240502 14:14:39.138285 15366 raft_consensus.cc:348] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:39.138487 15366 raft_consensus.cc:374] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.138549 15366 raft_consensus.cc:729] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.138715 15366 consensus_queue.cc:260] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:39.139127 15364 tablet_bootstrap.cc:654] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.139185 15366 ts_tablet_manager.cc:1398] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:39.146126 15364 tablet_bootstrap.cc:492] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:39.146245 15364 ts_tablet_manager.cc:1367] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20240502 14:14:39.146711 15364 raft_consensus.cc:348] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.146862 15364 raft_consensus.cc:374] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.146937 15364 raft_consensus.cc:729] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.147198 15364 consensus_queue.cc:260] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.147642 15364 ts_tablet_manager.cc:1398] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:14:39.147850 15364 tablet_bootstrap.cc:492] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:39.149021 15364 tablet_bootstrap.cc:654] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:39.150552 15364 tablet_bootstrap.cc:492] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:39.150674 15364 ts_tablet_manager.cc:1367] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:39.151165 15364 raft_consensus.cc:348] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.151286 15364 raft_consensus.cc:374] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:39.151372 15364 raft_consensus.cc:729] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:39.151540 15364 consensus_queue.cc:260] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.151909 15364 ts_tablet_manager.cc:1398] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:14:39.240378 14848 raft_consensus.cc:482] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:39.240526 14848 raft_consensus.cc:504] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:39.241170 14848 leader_election.cc:288] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:39.241541 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d39f694c8554638a3b10ebf7cd01350" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:39.241588 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d39f694c8554638a3b10ebf7cd01350" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:39.241722 13186 raft_consensus.cc:2454] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:39.241788 13318 raft_consensus.cc:2454] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:39.242127 13382 leader_election.cc:302] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:39.242381 14848 raft_consensus.cc:2781] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:39.242470 14848 raft_consensus.cc:482] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:39.242543 14848 raft_consensus.cc:3037] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.244288 14848 raft_consensus.cc:504] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:39.244794 14848 leader_election.cc:288] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:39.245195 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d39f694c8554638a3b10ebf7cd01350" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:39.245255 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d39f694c8554638a3b10ebf7cd01350" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:39.245390 13186 raft_consensus.cc:3037] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.245529 13318 raft_consensus.cc:3037] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.247678 13186 raft_consensus.cc:2454] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:39.248116 13382 leader_election.cc:302] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:39.248143 13318 raft_consensus.cc:2454] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:39.248399 14848 raft_consensus.cc:2781] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:39.248574 14848 raft_consensus.cc:686] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:39.248850 14848 consensus_queue.cc:237] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:39.251739 13036 catalog_manager.cc:5484] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:39.332031 14848 raft_consensus.cc:482] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:39.332201 14848 raft_consensus.cc:504] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.332768 14848 leader_election.cc:288] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:39.333101 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec7ae12c17247e9b8c03659435bd4e2" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:39.333370 13318 raft_consensus.cc:2454] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:39.333374 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec7ae12c17247e9b8c03659435bd4e2" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:39.333575 13580 raft_consensus.cc:2454] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:39.333838 13384 leader_election.cc:302] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:39.334102 14848 raft_consensus.cc:2781] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:39.334187 14848 raft_consensus.cc:482] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:39.334281 14848 raft_consensus.cc:3037] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.336223 14848 raft_consensus.cc:504] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.336707 14848 leader_election.cc:288] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:39.337087 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec7ae12c17247e9b8c03659435bd4e2" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:39.337143 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ec7ae12c17247e9b8c03659435bd4e2" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:39.337296 13318 raft_consensus.cc:3037] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.337363 13580 raft_consensus.cc:3037] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.339191 13580 raft_consensus.cc:2454] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:39.339270 13318 raft_consensus.cc:2454] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:39.339480 13382 leader_election.cc:302] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:39.339737 14848 raft_consensus.cc:2781] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:39.339924 14848 raft_consensus.cc:686] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:39.340186 14848 consensus_queue.cc:237] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.343011 13036 catalog_manager.cc:5484] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:39.406682 15274 raft_consensus.cc:482] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:39.406828 15274 raft_consensus.cc:504] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.407621 15274 leader_election.cc:288] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:39.407990 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5975e02205a43d5b93ed8ab51f8a0f8" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:39.408228 13449 raft_consensus.cc:2454] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:39.408192 15280 raft_consensus.cc:482] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:39.408416 15280 raft_consensus.cc:504] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.408711 13251 leader_election.cc:302] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:39.408959 15329 raft_consensus.cc:2781] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:39.409070 15329 raft_consensus.cc:482] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:39.409138 15280 leader_election.cc:288] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:39.409181 15329 raft_consensus.cc:3037] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.409576 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5975e02205a43d5b93ed8ab51f8a0f8" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:39.409685 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5975e02205a43d5b93ed8ab51f8a0f8" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:39.409786 13580 raft_consensus.cc:2454] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:39.410143 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5975e02205a43d5b93ed8ab51f8a0f8" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:39.410336 13449 raft_consensus.cc:2454] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:39.410642 13514 leader_election.cc:302] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:39.410857 15280 raft_consensus.cc:2781] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:39.410964 15280 raft_consensus.cc:482] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:39.411039 15280 raft_consensus.cc:3037] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.411793 15329 raft_consensus.cc:504] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.412365 15329 leader_election.cc:288] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:39.412565 13318 raft_consensus.cc:2379] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7eb4ecb2379b44c0954f2658c4a04350 in current term 1: Already voted for candidate cd67493f6f0142f896010f2b8a39887d in this term.
I20240502 14:14:39.412614 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5975e02205a43d5b93ed8ab51f8a0f8" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:39.412734 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5975e02205a43d5b93ed8ab51f8a0f8" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:39.412813 13449 raft_consensus.cc:3037] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:39.414656 13449 raft_consensus.cc:2454] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:39.414964 13251 leader_election.cc:302] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:39.415208 15329 raft_consensus.cc:2781] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:39.415365 15329 raft_consensus.cc:686] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:39.415578 15329 consensus_queue.cc:237] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.415761 15280 raft_consensus.cc:504] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:39.416105 13580 raft_consensus.cc:2379] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cd67493f6f0142f896010f2b8a39887d in current term 1: Already voted for candidate 7eb4ecb2379b44c0954f2658c4a04350 in this term.
I20240502 14:14:39.416491 15280 leader_election.cc:288] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:39.416707 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5975e02205a43d5b93ed8ab51f8a0f8" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:39.416678 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d5975e02205a43d5b93ed8ab51f8a0f8" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:39.416887 13448 raft_consensus.cc:2379] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 7eb4ecb2379b44c0954f2658c4a04350 in current term 1: Already voted for candidate cd67493f6f0142f896010f2b8a39887d in this term.
I20240502 14:14:39.417179 13514 leader_election.cc:302] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d
I20240502 14:14:39.417492 15280 raft_consensus.cc:2726] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:14:39.419886 13036 catalog_manager.cc:5484] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:39.679106 14848 consensus_queue.cc:1035] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:39.686717 15286 consensus_queue.cc:1035] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:39.779568 15375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:39.779827 15375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:39.787046 15375 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:39.805272 15329 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:39.812212 15274 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_13'
I20240502 14:14:39.844327 15286 consensus_queue.cc:1035] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:39.848460 14848 consensus_queue.cc:1035] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 244.083 ms
time per row: 0.122041 ms
I20240502 14:14:40.166265 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:40.174259 13561 tablet_service.cc:1467] Processing CreateTablet for tablet a187cc50f4cd45f1bcf841ea6f04a5e0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.174605 13168 tablet_service.cc:1467] Processing CreateTablet for tablet a187cc50f4cd45f1bcf841ea6f04a5e0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.174733 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a187cc50f4cd45f1bcf841ea6f04a5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.175118 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a187cc50f4cd45f1bcf841ea6f04a5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.175729 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 953692b0134743c980d43fcaf30238e1 (DEFAULT_TABLE table=rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.176131 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953692b0134743c980d43fcaf30238e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.176246 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 953692b0134743c980d43fcaf30238e1 (DEFAULT_TABLE table=rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.176636 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953692b0134743c980d43fcaf30238e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.177649 13167 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca719ad9a7b4ccea1c9db20434b92eb (DEFAULT_TABLE table=rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.177747 13298 tablet_service.cc:1467] Processing CreateTablet for tablet a187cc50f4cd45f1bcf841ea6f04a5e0 (DEFAULT_TABLE table=rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.178004 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca719ad9a7b4ccea1c9db20434b92eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.178160 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a187cc50f4cd45f1bcf841ea6f04a5e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.181519 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca719ad9a7b4ccea1c9db20434b92eb (DEFAULT_TABLE table=rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.181902 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca719ad9a7b4ccea1c9db20434b92eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.181843 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 953692b0134743c980d43fcaf30238e1 (DEFAULT_TABLE table=rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.182157 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 953692b0134743c980d43fcaf30238e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.185752 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca719ad9a7b4ccea1c9db20434b92eb (DEFAULT_TABLE table=rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:40.186240 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca719ad9a7b4ccea1c9db20434b92eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:40.194695 15440 tablet_bootstrap.cc:492] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:40.195364 15439 tablet_bootstrap.cc:492] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:40.196627 15440 tablet_bootstrap.cc:654] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.197778 15439 tablet_bootstrap.cc:654] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.203013 15439 tablet_bootstrap.cc:492] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:40.203140 15439 ts_tablet_manager.cc:1367] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240502 14:14:40.203648 15439 raft_consensus.cc:348] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:40.203786 15439 raft_consensus.cc:374] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.203877 15439 raft_consensus.cc:729] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.204090 15439 consensus_queue.cc:260] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:40.204646 15439 ts_tablet_manager.cc:1398] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:14:40.205817 15440 tablet_bootstrap.cc:492] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:40.205987 15440 ts_tablet_manager.cc:1367] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.007s
I20240502 14:14:40.206725 15440 raft_consensus.cc:348] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.206967 15440 raft_consensus.cc:374] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.207111 15440 raft_consensus.cc:729] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.208446 15440 consensus_queue.cc:260] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.209097 15440 ts_tablet_manager.cc:1398] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:40.209365 15440 tablet_bootstrap.cc:492] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:40.210825 15440 tablet_bootstrap.cc:654] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.214313 15440 tablet_bootstrap.cc:492] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:40.214526 15440 ts_tablet_manager.cc:1367] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240502 14:14:40.215390 15440 raft_consensus.cc:348] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.215657 15440 raft_consensus.cc:374] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.215763 15440 raft_consensus.cc:729] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.216028 15440 consensus_queue.cc:260] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.216177 15438 tablet_bootstrap.cc:492] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:40.216560 15444 tablet_bootstrap.cc:492] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:40.216650 15440 ts_tablet_manager.cc:1398] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:40.218432 15444 tablet_bootstrap.cc:654] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.218775 15438 tablet_bootstrap.cc:654] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.220220 15444 tablet_bootstrap.cc:492] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:40.220347 15444 ts_tablet_manager.cc:1367] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240502 14:14:40.221031 15444 raft_consensus.cc:348] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.221222 15444 raft_consensus.cc:374] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.221340 15444 raft_consensus.cc:729] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.221544 15444 consensus_queue.cc:260] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.222035 15444 ts_tablet_manager.cc:1398] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:40.222262 15444 tablet_bootstrap.cc:492] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:40.222316 15438 tablet_bootstrap.cc:492] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:40.222416 15438 ts_tablet_manager.cc:1367] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240502 14:14:40.223249 15438 raft_consensus.cc:348] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.223423 15438 raft_consensus.cc:374] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.223503 15438 raft_consensus.cc:729] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.223551 15444 tablet_bootstrap.cc:654] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.223701 15438 consensus_queue.cc:260] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.224320 15438 ts_tablet_manager.cc:1398] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:40.224733 15438 tablet_bootstrap.cc:492] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:40.226508 15444 tablet_bootstrap.cc:492] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:40.226660 15444 ts_tablet_manager.cc:1367] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:40.227185 15438 tablet_bootstrap.cc:654] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.227483 15444 raft_consensus.cc:348] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:40.227694 15444 raft_consensus.cc:374] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.227769 15444 raft_consensus.cc:729] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.228091 15444 consensus_queue.cc:260] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:40.228768 15444 ts_tablet_manager.cc:1398] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:40.229513 15444 tablet_bootstrap.cc:492] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:40.230237 15438 tablet_bootstrap.cc:492] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:40.230422 15438 ts_tablet_manager.cc:1367] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:14:40.230911 15444 tablet_bootstrap.cc:654] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.231145 15438 raft_consensus.cc:348] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.231483 15438 raft_consensus.cc:374] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.231595 15438 raft_consensus.cc:729] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.231868 15438 consensus_queue.cc:260] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.232576 15438 ts_tablet_manager.cc:1398] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:40.232802 15438 tablet_bootstrap.cc:492] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:40.233934 15444 tablet_bootstrap.cc:492] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:40.233939 15438 tablet_bootstrap.cc:654] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:40.234109 15444 ts_tablet_manager.cc:1367] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:14:40.234814 15444 raft_consensus.cc:348] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.235018 15444 raft_consensus.cc:374] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.235117 15444 raft_consensus.cc:729] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.235318 15444 consensus_queue.cc:260] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.235750 15438 tablet_bootstrap.cc:492] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:40.235929 15438 ts_tablet_manager.cc:1367] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:40.236598 15438 raft_consensus.cc:348] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:40.236822 15438 raft_consensus.cc:374] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:40.236877 15444 ts_tablet_manager.cc:1398] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:40.236968 15438 raft_consensus.cc:729] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:40.237206 15438 consensus_queue.cc:260] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:40.237712 15438 ts_tablet_manager.cc:1398] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:40.272540 15281 raft_consensus.cc:482] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:40.272696 15281 raft_consensus.cc:504] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:40.273545 15281 leader_election.cc:288] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:40.273820 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca719ad9a7b4ccea1c9db20434b92eb" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:40.274056 13187 raft_consensus.cc:2454] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:40.274037 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca719ad9a7b4ccea1c9db20434b92eb" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:40.274240 13448 raft_consensus.cc:2454] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:40.274485 13514 leader_election.cc:302] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:40.274997 15281 raft_consensus.cc:2781] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:40.275106 15281 raft_consensus.cc:482] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:40.275200 15281 raft_consensus.cc:3037] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.277124 15281 raft_consensus.cc:504] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:40.277664 15281 leader_election.cc:288] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:40.278146 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca719ad9a7b4ccea1c9db20434b92eb" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:40.278156 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca719ad9a7b4ccea1c9db20434b92eb" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:40.278314 13187 raft_consensus.cc:3037] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.278329 13448 raft_consensus.cc:3037] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.280922 13448 raft_consensus.cc:2454] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:40.281462 13514 leader_election.cc:302] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:40.282248 15281 raft_consensus.cc:2781] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:40.282529 13187 raft_consensus.cc:2454] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:40.282770 15281 raft_consensus.cc:686] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:40.283041 15281 consensus_queue.cc:237] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:40.286195 13039 catalog_manager.cc:5484] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:40.290657 15451 raft_consensus.cc:482] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:40.290797 15451 raft_consensus.cc:504] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.291380 15451 leader_election.cc:288] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:40.291662 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187cc50f4cd45f1bcf841ea6f04a5e0" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:40.291908 13580 raft_consensus.cc:2454] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:40.291925 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187cc50f4cd45f1bcf841ea6f04a5e0" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:40.292119 13319 raft_consensus.cc:2454] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:40.292258 13120 leader_election.cc:302] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:40.292734 15451 raft_consensus.cc:2781] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:40.292860 15451 raft_consensus.cc:482] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:40.292951 15451 raft_consensus.cc:3037] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.294617 15451 raft_consensus.cc:504] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.295090 15451 leader_election.cc:288] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:40.295334 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187cc50f4cd45f1bcf841ea6f04a5e0" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:40.295531 13580 raft_consensus.cc:3037] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.295635 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a187cc50f4cd45f1bcf841ea6f04a5e0" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:40.295794 13319 raft_consensus.cc:3037] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.297860 13319 raft_consensus.cc:2454] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:40.298115 13580 raft_consensus.cc:2454] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:40.298305 13122 leader_election.cc:302] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:40.298615 15451 raft_consensus.cc:2781] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:40.299053 15451 raft_consensus.cc:686] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:40.299333 15451 consensus_queue.cc:237] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.302174 13039 catalog_manager.cc:5484] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:40.401703 15274 raft_consensus.cc:482] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:40.401840 15274 raft_consensus.cc:504] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.402428 15274 leader_election.cc:288] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:40.402745 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953692b0134743c980d43fcaf30238e1" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:40.402812 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953692b0134743c980d43fcaf30238e1" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:40.402941 13581 raft_consensus.cc:2454] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:40.403043 13186 raft_consensus.cc:2454] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:40.403311 13251 leader_election.cc:302] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:40.403525 15274 raft_consensus.cc:2781] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:40.403647 15274 raft_consensus.cc:482] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:40.403739 15274 raft_consensus.cc:3037] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.405489 15274 raft_consensus.cc:504] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.405925 15274 leader_election.cc:288] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:40.406291 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953692b0134743c980d43fcaf30238e1" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:40.406340 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "953692b0134743c980d43fcaf30238e1" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:40.406461 13581 raft_consensus.cc:3037] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.406498 13186 raft_consensus.cc:3037] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:40.408282 13581 raft_consensus.cc:2454] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:40.408474 13186 raft_consensus.cc:2454] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:40.408697 13251 leader_election.cc:302] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:40.408932 15274 raft_consensus.cc:2781] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:40.409044 15274 raft_consensus.cc:686] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:40.409278 15274 consensus_queue.cc:237] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:40.412072 13039 catalog_manager.cc:5484] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } }
W20240502 14:14:40.746296 15455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:40.746542 15455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:40.753255 15455 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:40.775172 15451 consensus_queue.cc:1035] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_14'
I20240502 14:14:40.782574 15350 consensus_queue.cc:1035] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.785856 15280 consensus_queue.cc:1035] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.790172 15280 consensus_queue.cc:1035] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.998948 15274 consensus_queue.cc:1035] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:40.999558 15274 consensus_queue.cc:1035] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 239.702 ms
time per row: 0.119851 ms
I20240502 14:14:41.163870 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:41.171836 13298 tablet_service.cc:1467] Processing CreateTablet for tablet ba519bb3fb534e7fb6106f402e27446d (DEFAULT_TABLE table=rebalance_test_table_15 [id=557f9d603bda4eb385d26221f3612c33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.171965 13166 tablet_service.cc:1467] Processing CreateTablet for tablet ba519bb3fb534e7fb6106f402e27446d (DEFAULT_TABLE table=rebalance_test_table_15 [id=557f9d603bda4eb385d26221f3612c33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.172289 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba519bb3fb534e7fb6106f402e27446d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.172305 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba519bb3fb534e7fb6106f402e27446d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.173945 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 9a95b9049a344b98be91b9961f3e5bee (DEFAULT_TABLE table=rebalance_test_table_15 [id=557f9d603bda4eb385d26221f3612c33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.174309 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a95b9049a344b98be91b9961f3e5bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.174460 13561 tablet_service.cc:1467] Processing CreateTablet for tablet 9a95b9049a344b98be91b9961f3e5bee (DEFAULT_TABLE table=rebalance_test_table_15 [id=557f9d603bda4eb385d26221f3612c33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.174818 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a95b9049a344b98be91b9961f3e5bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.175258 13428 tablet_service.cc:1467] Processing CreateTablet for tablet 9a95b9049a344b98be91b9961f3e5bee (DEFAULT_TABLE table=rebalance_test_table_15 [id=557f9d603bda4eb385d26221f3612c33]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.175629 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a95b9049a344b98be91b9961f3e5bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.176872 13167 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad2eb265e0f469787eb1699e339e0ed (DEFAULT_TABLE table=rebalance_test_table_15 [id=557f9d603bda4eb385d26221f3612c33]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.177192 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad2eb265e0f469787eb1699e339e0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.177569 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad2eb265e0f469787eb1699e339e0ed (DEFAULT_TABLE table=rebalance_test_table_15 [id=557f9d603bda4eb385d26221f3612c33]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.177914 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad2eb265e0f469787eb1699e339e0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.179934 13427 tablet_service.cc:1467] Processing CreateTablet for tablet ba519bb3fb534e7fb6106f402e27446d (DEFAULT_TABLE table=rebalance_test_table_15 [id=557f9d603bda4eb385d26221f3612c33]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.180289 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba519bb3fb534e7fb6106f402e27446d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.181589 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad2eb265e0f469787eb1699e339e0ed (DEFAULT_TABLE table=rebalance_test_table_15 [id=557f9d603bda4eb385d26221f3612c33]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:41.181937 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad2eb265e0f469787eb1699e339e0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:41.188050 15518 tablet_bootstrap.cc:492] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:41.192102 15518 tablet_bootstrap.cc:654] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.193018 15520 tablet_bootstrap.cc:492] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:41.194186 15521 tablet_bootstrap.cc:492] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:41.195117 15520 tablet_bootstrap.cc:654] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.195811 15521 tablet_bootstrap.cc:654] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.196977 15519 tablet_bootstrap.cc:492] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:41.197957 15520 tablet_bootstrap.cc:492] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:41.198213 15520 ts_tablet_manager.cc:1367] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:14:41.198913 15519 tablet_bootstrap.cc:654] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.199229 15521 tablet_bootstrap.cc:492] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:41.199204 15520 raft_consensus.cc:348] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:41.199360 15521 ts_tablet_manager.cc:1367] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:14:41.199540 15520 raft_consensus.cc:374] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.199681 15520 raft_consensus.cc:729] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.200040 15520 consensus_queue.cc:260] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:41.200882 15520 ts_tablet_manager.cc:1398] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:41.201293 15520 tablet_bootstrap.cc:492] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:41.201313 15521 raft_consensus.cc:348] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:41.201591 15521 raft_consensus.cc:374] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.201714 15521 raft_consensus.cc:729] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.202267 15521 consensus_queue.cc:260] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:41.202744 15520 tablet_bootstrap.cc:654] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.203457 15521 ts_tablet_manager.cc:1398] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:14:41.203733 15521 tablet_bootstrap.cc:492] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:41.205600 15521 tablet_bootstrap.cc:654] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.207610 15518 tablet_bootstrap.cc:492] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:41.207849 15518 ts_tablet_manager.cc:1367] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.003s
I20240502 14:14:41.208875 15518 raft_consensus.cc:348] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:41.209105 15518 raft_consensus.cc:374] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.209208 15518 raft_consensus.cc:729] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.209633 15518 consensus_queue.cc:260] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:41.210211 15518 ts_tablet_manager.cc:1398] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:41.210211 15519 tablet_bootstrap.cc:492] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:41.210387 15519 ts_tablet_manager.cc:1367] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240502 14:14:41.210498 15518 tablet_bootstrap.cc:492] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:41.211500 15521 tablet_bootstrap.cc:492] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:41.211665 15521 ts_tablet_manager.cc:1367] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.002s
I20240502 14:14:41.211920 15518 tablet_bootstrap.cc:654] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.212415 15521 raft_consensus.cc:348] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.212622 15521 raft_consensus.cc:374] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.212731 15521 raft_consensus.cc:729] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.213644 15519 raft_consensus.cc:348] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.213954 15519 raft_consensus.cc:374] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.214109 15519 raft_consensus.cc:729] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.214501 15521 consensus_queue.cc:260] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.216293 15521 ts_tablet_manager.cc:1398] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:14:41.216567 15520 tablet_bootstrap.cc:492] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:41.216712 15520 ts_tablet_manager.cc:1367] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.005s
I20240502 14:14:41.217383 15520 raft_consensus.cc:348] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.217629 15520 raft_consensus.cc:374] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.217710 15520 raft_consensus.cc:729] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.217937 15520 consensus_queue.cc:260] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.214421 15519 consensus_queue.cc:260] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.222077 15519 ts_tablet_manager.cc:1398] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:14:41.222194 15520 ts_tablet_manager.cc:1398] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:41.222412 15519 tablet_bootstrap.cc:492] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:41.222447 15520 tablet_bootstrap.cc:492] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:41.224963 15519 tablet_bootstrap.cc:654] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.228525 15520 tablet_bootstrap.cc:654] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:41.230429 15519 tablet_bootstrap.cc:492] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:41.230551 15518 tablet_bootstrap.cc:492] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:41.230576 15519 ts_tablet_manager.cc:1367] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240502 14:14:41.230687 15518 ts_tablet_manager.cc:1367] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.011s
I20240502 14:14:41.230918 15520 tablet_bootstrap.cc:492] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:41.231045 15520 ts_tablet_manager.cc:1367] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20240502 14:14:41.231236 15518 raft_consensus.cc:348] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.231232 15519 raft_consensus.cc:348] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.231385 15518 raft_consensus.cc:374] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.231402 15519 raft_consensus.cc:374] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.231452 15518 raft_consensus.cc:729] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.231467 15519 raft_consensus.cc:729] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.231657 15518 consensus_queue.cc:260] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.231658 15520 raft_consensus.cc:348] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.231688 15519 consensus_queue.cc:260] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.231825 15520 raft_consensus.cc:374] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:41.232102 15518 ts_tablet_manager.cc:1398] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:14:41.232123 15519 ts_tablet_manager.cc:1398] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 14:14:41.232591 15520 raft_consensus.cc:729] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:41.232815 15520 consensus_queue.cc:260] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.233390 15520 ts_tablet_manager.cc:1398] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:41.346259 15513 raft_consensus.cc:482] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:41.346401 15513 raft_consensus.cc:504] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:41.346894 15513 leader_election.cc:288] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:41.347327 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a95b9049a344b98be91b9961f3e5bee" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:41.347349 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a95b9049a344b98be91b9961f3e5bee" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:41.347577 13580 raft_consensus.cc:2454] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:41.347577 13449 raft_consensus.cc:2454] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:41.347975 13251 leader_election.cc:302] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:41.348210 15513 raft_consensus.cc:2781] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:41.348320 15513 raft_consensus.cc:482] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:41.348407 15513 raft_consensus.cc:3037] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.350127 15513 raft_consensus.cc:504] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:41.350548 15513 leader_election.cc:288] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:41.350945 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a95b9049a344b98be91b9961f3e5bee" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:41.351038 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a95b9049a344b98be91b9961f3e5bee" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:41.351116 13580 raft_consensus.cc:3037] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.351199 13449 raft_consensus.cc:3037] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.353579 13580 raft_consensus.cc:2454] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:41.353591 13449 raft_consensus.cc:2454] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:41.353942 13251 leader_election.cc:302] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:41.354172 15513 raft_consensus.cc:2781] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:41.354292 15513 raft_consensus.cc:686] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:41.354540 15513 consensus_queue.cc:237] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:41.357625 13038 catalog_manager.cc:5484] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:41.362329 15274 raft_consensus.cc:482] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:41.362452 15274 raft_consensus.cc:504] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.362931 15274 leader_election.cc:288] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:41.363559 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba519bb3fb534e7fb6106f402e27446d" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:41.363570 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba519bb3fb534e7fb6106f402e27446d" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:41.363745 13449 raft_consensus.cc:2454] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:41.363759 13187 raft_consensus.cc:2454] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:41.364275 13251 leader_election.cc:302] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:41.364462 15274 raft_consensus.cc:2781] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:41.364531 15274 raft_consensus.cc:482] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:41.364629 15274 raft_consensus.cc:3037] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.366142 15274 raft_consensus.cc:504] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.366600 15274 leader_election.cc:288] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:41.366940 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba519bb3fb534e7fb6106f402e27446d" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:41.366981 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba519bb3fb534e7fb6106f402e27446d" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:41.367120 13187 raft_consensus.cc:3037] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.367137 13449 raft_consensus.cc:3037] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.369484 13187 raft_consensus.cc:2454] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:41.370103 13251 leader_election.cc:302] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:41.372130 13449 raft_consensus.cc:2454] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:41.373337 15274 raft_consensus.cc:2781] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:41.373462 15274 raft_consensus.cc:686] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:41.373672 15274 consensus_queue.cc:237] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.376407 13042 catalog_manager.cc:5484] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:41.382546 15478 raft_consensus.cc:482] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:41.382668 15478 raft_consensus.cc:504] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.383245 15478 leader_election.cc:288] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:41.383606 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad2eb265e0f469787eb1699e339e0ed" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:41.383615 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad2eb265e0f469787eb1699e339e0ed" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:41.383762 13187 raft_consensus.cc:2454] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:41.383754 13449 raft_consensus.cc:2454] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:41.384076 13514 leader_election.cc:302] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:41.384305 15478 raft_consensus.cc:2781] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:41.384403 15478 raft_consensus.cc:482] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:41.384475 15478 raft_consensus.cc:3037] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.386054 15478 raft_consensus.cc:504] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.386492 15478 leader_election.cc:288] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:41.386837 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad2eb265e0f469787eb1699e339e0ed" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:41.386926 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad2eb265e0f469787eb1699e339e0ed" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:41.387010 13187 raft_consensus.cc:3037] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.387070 13449 raft_consensus.cc:3037] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:41.389655 13187 raft_consensus.cc:2454] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:41.389655 13449 raft_consensus.cc:2454] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:41.389952 13514 leader_election.cc:302] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:41.390187 15478 raft_consensus.cc:2781] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:41.390324 15478 raft_consensus.cc:686] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:41.390524 15478 consensus_queue.cc:237] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:41.392844 13042 catalog_manager.cc:5484] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:41.737553 15535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:41.737888 15535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:41.744848 15535 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:41.760504 15274 consensus_queue.cc:1035] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:41.767248 15513 consensus_queue.cc:1035] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:41.772961 15478 consensus_queue.cc:1035] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_15'
I20240502 14:14:41.786770 15478 consensus_queue.cc:1035] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:41.831821 15548 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:41.835945 15274 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 300.652 ms
time per row: 0.150326 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:14:43.107697 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:43.124131 13167 tablet_service.cc:1467] Processing CreateTablet for tablet bd170e9091a84185ae8fa6714110d9f3 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.124194 13296 tablet_service.cc:1467] Processing CreateTablet for tablet bd170e9091a84185ae8fa6714110d9f3 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.124600 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd170e9091a84185ae8fa6714110d9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.124647 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd170e9091a84185ae8fa6714110d9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.125622 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 624672f7de7843a595be3f75203f65e6 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.125699 13428 tablet_service.cc:1467] Processing CreateTablet for tablet bd170e9091a84185ae8fa6714110d9f3 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.126020 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 624672f7de7843a595be3f75203f65e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.126113 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd170e9091a84185ae8fa6714110d9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.126163 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 624672f7de7843a595be3f75203f65e6 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.126547 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 624672f7de7843a595be3f75203f65e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.129354 13561 tablet_service.cc:1467] Processing CreateTablet for tablet bbd12f1c9884497b82c941b7231645bb (DEFAULT_TABLE table=rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.129765 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd12f1c9884497b82c941b7231645bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.133890 13166 tablet_service.cc:1467] Processing CreateTablet for tablet bbd12f1c9884497b82c941b7231645bb (DEFAULT_TABLE table=rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.134305 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd12f1c9884497b82c941b7231645bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.134557 13298 tablet_service.cc:1467] Processing CreateTablet for tablet bbd12f1c9884497b82c941b7231645bb (DEFAULT_TABLE table=rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.134943 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd12f1c9884497b82c941b7231645bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.135291 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 624672f7de7843a595be3f75203f65e6 (DEFAULT_TABLE table=rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:43.139925 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 624672f7de7843a595be3f75203f65e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:43.142297 15607 tablet_bootstrap.cc:492] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:43.144312 15607 tablet_bootstrap.cc:654] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.147353 15608 tablet_bootstrap.cc:492] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:43.149082 15608 tablet_bootstrap.cc:654] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.163318 15607 tablet_bootstrap.cc:492] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:43.163570 15607 ts_tablet_manager.cc:1367] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.016s
I20240502 14:14:43.163851 15612 tablet_bootstrap.cc:492] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:43.164474 15607 raft_consensus.cc:348] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:43.164814 15607 raft_consensus.cc:374] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.164973 15607 raft_consensus.cc:729] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.165320 15607 consensus_queue.cc:260] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:43.165984 15612 tablet_bootstrap.cc:654] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.166704 15607 ts_tablet_manager.cc:1398] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:43.167301 15606 tablet_bootstrap.cc:492] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:43.169191 15607 tablet_bootstrap.cc:492] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:43.169260 15606 tablet_bootstrap.cc:654] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.170761 15607 tablet_bootstrap.cc:654] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.170953 15612 tablet_bootstrap.cc:492] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:43.171129 15612 ts_tablet_manager.cc:1367] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:14:43.172047 15612 raft_consensus.cc:348] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:43.172324 15612 raft_consensus.cc:374] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.172462 15612 raft_consensus.cc:729] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.172725 15612 consensus_queue.cc:260] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:43.173373 15612 ts_tablet_manager.cc:1398] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:43.173676 15612 tablet_bootstrap.cc:492] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:43.176091 15608 tablet_bootstrap.cc:492] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:43.176297 15608 ts_tablet_manager.cc:1367] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.016s
I20240502 14:14:43.176304 15612 tablet_bootstrap.cc:654] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.177076 15608 raft_consensus.cc:348] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:43.177376 15608 raft_consensus.cc:374] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.178170 15606 tablet_bootstrap.cc:492] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:43.178331 15608 raft_consensus.cc:729] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.178362 15606 ts_tablet_manager.cc:1367] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 14:14:43.178645 15608 consensus_queue.cc:260] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:43.179281 15608 ts_tablet_manager.cc:1398] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:43.179363 15606 raft_consensus.cc:348] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:43.179682 15606 raft_consensus.cc:374] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.179860 15606 raft_consensus.cc:729] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.179926 15608 tablet_bootstrap.cc:492] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:43.180255 15606 consensus_queue.cc:260] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:43.181461 15608 tablet_bootstrap.cc:654] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.182037 15606 ts_tablet_manager.cc:1398] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:43.182338 15606 tablet_bootstrap.cc:492] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:43.183826 15606 tablet_bootstrap.cc:654] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.189170 15612 tablet_bootstrap.cc:492] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:43.189433 15612 ts_tablet_manager.cc:1367] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20240502 14:14:43.189837 15608 tablet_bootstrap.cc:492] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:43.190122 15608 ts_tablet_manager.cc:1367] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240502 14:14:43.190289 15612 raft_consensus.cc:348] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:43.190498 15612 raft_consensus.cc:374] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.190578 15612 raft_consensus.cc:729] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.190768 15608 raft_consensus.cc:348] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:43.190804 15612 consensus_queue.cc:260] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:43.191078 15608 raft_consensus.cc:374] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.192059 15608 raft_consensus.cc:729] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.192341 15612 ts_tablet_manager.cc:1398] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:43.192287 15608 consensus_queue.cc:260] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:43.192612 15612 tablet_bootstrap.cc:492] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:43.192821 15608 ts_tablet_manager.cc:1398] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:43.194108 15612 tablet_bootstrap.cc:654] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:43.195654 15606 tablet_bootstrap.cc:492] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:43.196619 15606 ts_tablet_manager.cc:1367] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 14:14:43.197389 15606 raft_consensus.cc:348] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:43.197628 15606 raft_consensus.cc:374] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.197736 15606 raft_consensus.cc:729] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.198038 15606 consensus_queue.cc:260] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:43.198770 15612 tablet_bootstrap.cc:492] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:43.198964 15612 ts_tablet_manager.cc:1367] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.002s
I20240502 14:14:43.199546 15606 ts_tablet_manager.cc:1398] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:43.199592 15607 tablet_bootstrap.cc:492] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:43.199786 15607 ts_tablet_manager.cc:1367] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.000s
I20240502 14:14:43.199824 15612 raft_consensus.cc:348] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:43.200032 15612 raft_consensus.cc:374] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.200129 15612 raft_consensus.cc:729] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.200333 15612 consensus_queue.cc:260] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:43.200577 15607 raft_consensus.cc:348] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:43.200858 15607 raft_consensus.cc:374] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:43.200887 15612 ts_tablet_manager.cc:1398] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:43.200987 15607 raft_consensus.cc:729] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:43.201210 15607 consensus_queue.cc:260] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:43.202500 15607 ts_tablet_manager.cc:1398] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:14:43.272845 15274 raft_consensus.cc:482] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:43.273025 15274 raft_consensus.cc:504] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:43.273626 15274 leader_election.cc:288] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:43.273983 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "624672f7de7843a595be3f75203f65e6" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:43.274201 13447 raft_consensus.cc:2454] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:43.274168 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "624672f7de7843a595be3f75203f65e6" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:43.274360 13581 raft_consensus.cc:2454] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:43.274675 13251 leader_election.cc:302] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:43.274932 15274 raft_consensus.cc:2781] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:43.275051 15274 raft_consensus.cc:482] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:43.275152 15274 raft_consensus.cc:3037] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.277729 15274 raft_consensus.cc:504] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:43.278295 15274 leader_election.cc:288] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:43.278663 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "624672f7de7843a595be3f75203f65e6" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:43.278869 13447 raft_consensus.cc:3037] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.278899 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "624672f7de7843a595be3f75203f65e6" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:43.279075 13581 raft_consensus.cc:3037] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.281548 13447 raft_consensus.cc:2454] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:43.281666 13581 raft_consensus.cc:2454] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:43.281998 13251 leader_election.cc:302] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:43.282317 15274 raft_consensus.cc:2781] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:43.282519 15274 raft_consensus.cc:686] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:43.282827 15274 consensus_queue.cc:237] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:43.286152 13036 catalog_manager.cc:5484] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:43.315054 15534 raft_consensus.cc:482] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:43.315214 15534 raft_consensus.cc:504] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:43.315821 15534 leader_election.cc:288] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:43.316277 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd170e9091a84185ae8fa6714110d9f3" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:43.316314 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd170e9091a84185ae8fa6714110d9f3" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:43.316519 13318 raft_consensus.cc:2454] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:43.316540 13187 raft_consensus.cc:2454] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:43.317008 13384 leader_election.cc:302] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:43.317364 15534 raft_consensus.cc:2781] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:43.317507 15534 raft_consensus.cc:482] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:43.317615 15534 raft_consensus.cc:3037] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.319875 15534 raft_consensus.cc:504] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:43.320550 15534 leader_election.cc:288] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:43.321055 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd170e9091a84185ae8fa6714110d9f3" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:43.321173 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd170e9091a84185ae8fa6714110d9f3" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:43.321287 13318 raft_consensus.cc:3037] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.321413 13187 raft_consensus.cc:3037] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.324038 13187 raft_consensus.cc:2454] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:43.324045 13318 raft_consensus.cc:2454] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:43.324519 13384 leader_election.cc:302] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:43.324791 15534 raft_consensus.cc:2781] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:43.324914 15534 raft_consensus.cc:686] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:43.325176 15534 consensus_queue.cc:237] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:43.328291 13036 catalog_manager.cc:5484] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:43.341629 15281 raft_consensus.cc:482] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:43.341809 15281 raft_consensus.cc:504] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:43.342386 15281 leader_election.cc:288] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:43.342774 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd12f1c9884497b82c941b7231645bb" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:43.342855 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd12f1c9884497b82c941b7231645bb" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:43.342998 13187 raft_consensus.cc:2454] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:43.343084 13318 raft_consensus.cc:2454] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:43.343425 13514 leader_election.cc:302] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:43.343672 15281 raft_consensus.cc:2781] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:43.343753 15281 raft_consensus.cc:482] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:43.343860 15281 raft_consensus.cc:3037] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.345588 15281 raft_consensus.cc:504] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:43.346086 15281 leader_election.cc:288] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:43.346374 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd12f1c9884497b82c941b7231645bb" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:43.346537 13187 raft_consensus.cc:3037] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.346458 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbd12f1c9884497b82c941b7231645bb" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:43.346666 13318 raft_consensus.cc:3037] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:43.348403 13187 raft_consensus.cc:2454] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:43.348412 13318 raft_consensus.cc:2454] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:43.348764 13516 leader_election.cc:302] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:43.349004 15281 raft_consensus.cc:2781] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:43.349150 15281 raft_consensus.cc:686] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:43.349388 15281 consensus_queue.cc:237] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:43.352173 13036 catalog_manager.cc:5484] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } }
W20240502 14:14:43.692449 15623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:43.692731 15623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:43.699606 15623 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:43.725759 15274 consensus_queue.cc:1035] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:43.732569 15601 consensus_queue.cc:1035] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_16'
I20240502 14:14:43.737558 15281 consensus_queue.cc:1035] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:43.743566 15281 consensus_queue.cc:1035] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:43.833700 13187 raft_consensus.cc:1261] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:43.834007 13318 raft_consensus.cc:1261] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:43.834295 15534 consensus_queue.cc:1035] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:43.834609 15609 consensus_queue.cc:1035] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 221.385 ms
time per row: 0.110693 ms
I20240502 14:14:44.093295 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:44.101712 13559 tablet_service.cc:1467] Processing CreateTablet for tablet ab51b049c3674b629b55ed29df5b1fab (DEFAULT_TABLE table=rebalance_test_table_17 [id=cbc2cd67f3c24195825b542cf2878195]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.101922 13167 tablet_service.cc:1467] Processing CreateTablet for tablet ab51b049c3674b629b55ed29df5b1fab (DEFAULT_TABLE table=rebalance_test_table_17 [id=cbc2cd67f3c24195825b542cf2878195]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.102155 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab51b049c3674b629b55ed29df5b1fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.102301 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab51b049c3674b629b55ed29df5b1fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.103106 13296 tablet_service.cc:1467] Processing CreateTablet for tablet c3e9160c46d64fa0a964cc39aec91dfd (DEFAULT_TABLE table=rebalance_test_table_17 [id=cbc2cd67f3c24195825b542cf2878195]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.103269 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 766f50e1e5a44c8ba211a446bcf86705 (DEFAULT_TABLE table=rebalance_test_table_17 [id=cbc2cd67f3c24195825b542cf2878195]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.103477 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e9160c46d64fa0a964cc39aec91dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.103616 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766f50e1e5a44c8ba211a446bcf86705. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.105537 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 766f50e1e5a44c8ba211a446bcf86705 (DEFAULT_TABLE table=rebalance_test_table_17 [id=cbc2cd67f3c24195825b542cf2878195]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.105885 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766f50e1e5a44c8ba211a446bcf86705. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.107275 13428 tablet_service.cc:1467] Processing CreateTablet for tablet ab51b049c3674b629b55ed29df5b1fab (DEFAULT_TABLE table=rebalance_test_table_17 [id=cbc2cd67f3c24195825b542cf2878195]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.107601 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab51b049c3674b629b55ed29df5b1fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.109287 13427 tablet_service.cc:1467] Processing CreateTablet for tablet c3e9160c46d64fa0a964cc39aec91dfd (DEFAULT_TABLE table=rebalance_test_table_17 [id=cbc2cd67f3c24195825b542cf2878195]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.109616 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e9160c46d64fa0a964cc39aec91dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.109892 13561 tablet_service.cc:1467] Processing CreateTablet for tablet 766f50e1e5a44c8ba211a446bcf86705 (DEFAULT_TABLE table=rebalance_test_table_17 [id=cbc2cd67f3c24195825b542cf2878195]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.110340 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766f50e1e5a44c8ba211a446bcf86705. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.114248 13168 tablet_service.cc:1467] Processing CreateTablet for tablet c3e9160c46d64fa0a964cc39aec91dfd (DEFAULT_TABLE table=rebalance_test_table_17 [id=cbc2cd67f3c24195825b542cf2878195]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:44.114579 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e9160c46d64fa0a964cc39aec91dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:44.132227 15692 tablet_bootstrap.cc:492] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:44.133358 15691 tablet_bootstrap.cc:492] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:44.134088 15692 tablet_bootstrap.cc:654] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.135982 15691 tablet_bootstrap.cc:654] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.136289 15693 tablet_bootstrap.cc:492] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:44.138386 15693 tablet_bootstrap.cc:654] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.141011 15694 tablet_bootstrap.cc:492] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:44.141883 15691 tablet_bootstrap.cc:492] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:44.142074 15691 ts_tablet_manager.cc:1367] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20240502 14:14:44.142877 15691 raft_consensus.cc:348] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.143103 15691 raft_consensus.cc:374] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.143208 15691 raft_consensus.cc:729] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.143447 15691 consensus_queue.cc:260] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.143775 15694 tablet_bootstrap.cc:654] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.144026 15691 ts_tablet_manager.cc:1398] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:44.149992 15693 tablet_bootstrap.cc:492] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:44.150978 15693 ts_tablet_manager.cc:1367] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240502 14:14:44.151865 15693 raft_consensus.cc:348] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.152101 15693 raft_consensus.cc:374] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.152207 15693 raft_consensus.cc:729] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.152484 15693 consensus_queue.cc:260] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.153213 15693 ts_tablet_manager.cc:1398] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:44.153976 15693 tablet_bootstrap.cc:492] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:44.154291 15692 tablet_bootstrap.cc:492] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:44.154453 15692 ts_tablet_manager.cc:1367] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.014s
I20240502 14:14:44.155287 15693 tablet_bootstrap.cc:654] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.155165 15692 raft_consensus.cc:348] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.155380 15692 raft_consensus.cc:374] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.155491 15692 raft_consensus.cc:729] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.155766 15692 consensus_queue.cc:260] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.156417 15692 ts_tablet_manager.cc:1398] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:44.157006 15694 tablet_bootstrap.cc:492] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:44.157136 15692 tablet_bootstrap.cc:492] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:44.157130 15694 ts_tablet_manager.cc:1367] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240502 14:14:44.157913 15694 raft_consensus.cc:348] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.158123 15694 raft_consensus.cc:374] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.158205 15694 raft_consensus.cc:729] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.158447 15694 consensus_queue.cc:260] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.158659 15692 tablet_bootstrap.cc:654] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.159080 15694 ts_tablet_manager.cc:1398] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:44.159327 15694 tablet_bootstrap.cc:492] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:44.160454 15692 tablet_bootstrap.cc:492] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:44.160599 15692 ts_tablet_manager.cc:1367] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:44.160801 15694 tablet_bootstrap.cc:654] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.161345 15692 raft_consensus.cc:348] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.161521 15692 raft_consensus.cc:374] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.161604 15692 raft_consensus.cc:729] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.161871 15692 consensus_queue.cc:260] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.162340 15692 ts_tablet_manager.cc:1398] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:44.166059 15694 tablet_bootstrap.cc:492] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:44.166059 15693 tablet_bootstrap.cc:492] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:44.166200 15693 ts_tablet_manager.cc:1367] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:14:44.166245 15694 ts_tablet_manager.cc:1367] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240502 14:14:44.166889 15694 raft_consensus.cc:348] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.167022 15693 raft_consensus.cc:348] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.167135 15694 raft_consensus.cc:374] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.167245 15693 raft_consensus.cc:374] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.167243 15694 raft_consensus.cc:729] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.167352 15693 raft_consensus.cc:729] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.167462 15694 consensus_queue.cc:260] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.167593 15693 consensus_queue.cc:260] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.168112 15694 ts_tablet_manager.cc:1398] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:44.168581 15694 tablet_bootstrap.cc:492] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:44.169947 15694 tablet_bootstrap.cc:654] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.169970 15693 ts_tablet_manager.cc:1398] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:14:44.177846 15693 tablet_bootstrap.cc:492] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:44.179908 15693 tablet_bootstrap.cc:654] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:44.187973 15693 tablet_bootstrap.cc:492] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:44.188123 15693 ts_tablet_manager.cc:1367] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240502 14:14:44.188627 15694 tablet_bootstrap.cc:492] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:44.188774 15694 ts_tablet_manager.cc:1367] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.017s
I20240502 14:14:44.189409 15693 raft_consensus.cc:348] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.189463 15694 raft_consensus.cc:348] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.189597 15693 raft_consensus.cc:374] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.189630 15694 raft_consensus.cc:374] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:44.189710 15694 raft_consensus.cc:729] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.189749 15693 raft_consensus.cc:729] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:44.189956 15693 consensus_queue.cc:260] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.189939 15694 consensus_queue.cc:260] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.190454 15694 ts_tablet_manager.cc:1398] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:44.191231 15693 ts_tablet_manager.cc:1398] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:44.194559 15609 raft_consensus.cc:482] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:44.194715 15609 raft_consensus.cc:504] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.195355 15609 leader_election.cc:288] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:44.195775 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e9160c46d64fa0a964cc39aec91dfd" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:44.195946 13317 raft_consensus.cc:2454] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:44.196206 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e9160c46d64fa0a964cc39aec91dfd" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:44.196274 13384 leader_election.cc:302] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:44.196382 13186 raft_consensus.cc:2454] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:44.196571 15609 raft_consensus.cc:2781] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:44.196689 15609 raft_consensus.cc:482] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:44.196807 15609 raft_consensus.cc:3037] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.198834 15609 raft_consensus.cc:504] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.199399 15609 leader_election.cc:288] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:44.199687 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e9160c46d64fa0a964cc39aec91dfd" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:44.199729 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e9160c46d64fa0a964cc39aec91dfd" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:44.199875 13186 raft_consensus.cc:3037] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.199901 13317 raft_consensus.cc:3037] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.202569 13317 raft_consensus.cc:2454] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:44.202569 13186 raft_consensus.cc:2454] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:44.202970 13384 leader_election.cc:302] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:44.203358 15609 raft_consensus.cc:2781] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:44.203521 15609 raft_consensus.cc:686] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:44.203775 15609 consensus_queue.cc:237] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.206254 13040 catalog_manager.cc:5484] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:44.208523 15609 raft_consensus.cc:482] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:44.208678 15609 raft_consensus.cc:504] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.209498 15609 leader_election.cc:288] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:44.209574 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab51b049c3674b629b55ed29df5b1fab" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:44.209604 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab51b049c3674b629b55ed29df5b1fab" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:44.209774 13186 raft_consensus.cc:2454] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:44.209784 13581 raft_consensus.cc:2454] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:44.210136 13382 leader_election.cc:302] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:44.210394 15534 raft_consensus.cc:2781] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:44.210513 15534 raft_consensus.cc:482] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:44.210620 15534 raft_consensus.cc:3037] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.212412 15534 raft_consensus.cc:504] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.212785 15534 leader_election.cc:288] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:44.213115 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab51b049c3674b629b55ed29df5b1fab" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:44.213285 13581 raft_consensus.cc:3037] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.214799 13581 raft_consensus.cc:2454] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:44.215068 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab51b049c3674b629b55ed29df5b1fab" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:44.215137 13382 leader_election.cc:302] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:44.215212 13186 raft_consensus.cc:3037] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.215381 15534 raft_consensus.cc:2781] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:44.215521 15534 raft_consensus.cc:686] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:44.215736 15534 consensus_queue.cc:237] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.217165 13186 raft_consensus.cc:2454] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:44.218616 13040 catalog_manager.cc:5484] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:44.253022 15666 raft_consensus.cc:482] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:44.253183 15666 raft_consensus.cc:504] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.253813 15666 leader_election.cc:288] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:44.254251 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766f50e1e5a44c8ba211a446bcf86705" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:44.254236 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766f50e1e5a44c8ba211a446bcf86705" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:44.254446 13186 raft_consensus.cc:2454] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:44.254460 13448 raft_consensus.cc:2454] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:44.254781 13514 leader_election.cc:302] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:44.254990 15666 raft_consensus.cc:2781] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:44.255096 15666 raft_consensus.cc:482] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:44.255179 15666 raft_consensus.cc:3037] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.257115 15666 raft_consensus.cc:504] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.257584 15666 leader_election.cc:288] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:44.257969 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766f50e1e5a44c8ba211a446bcf86705" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:44.258019 13448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "766f50e1e5a44c8ba211a446bcf86705" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:44.258150 13186 raft_consensus.cc:3037] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.258191 13448 raft_consensus.cc:3037] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:44.259936 13448 raft_consensus.cc:2454] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:44.259968 13186 raft_consensus.cc:2454] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:44.260310 13514 leader_election.cc:302] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:44.260540 15666 raft_consensus.cc:2781] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:44.260669 15666 raft_consensus.cc:686] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:44.260913 15666 consensus_queue.cc:237] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:44.263667 13040 catalog_manager.cc:5484] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:44.595204 15707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:44.595499 15707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:44.603206 15707 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:14:44.605341 15534 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:44.620784 15609 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:44.630280 15609 consensus_queue.cc:1035] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:44.638090 15534 consensus_queue.cc:1035] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:44.660668 15666 consensus_queue.cc:1035] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_17'
I20240502 14:14:44.664440 15281 consensus_queue.cc:1035] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 205.418 ms
time per row: 0.102709 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:14:45.876359 13042 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:45.888741 13296 tablet_service.cc:1467] Processing CreateTablet for tablet b3b57323bd14448b9406ce53ada769e5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.888988 13166 tablet_service.cc:1467] Processing CreateTablet for tablet b3b57323bd14448b9406ce53ada769e5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.888984 13559 tablet_service.cc:1467] Processing CreateTablet for tablet b3b57323bd14448b9406ce53ada769e5 (DEFAULT_TABLE table=rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.889264 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b57323bd14448b9406ce53ada769e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.889431 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b57323bd14448b9406ce53ada769e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.889432 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3b57323bd14448b9406ce53ada769e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.894114 15776 tablet_bootstrap.cc:492] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:45.894843 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 718223dd8fd3432cb174a53325dc784c (DEFAULT_TABLE table=rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.895220 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718223dd8fd3432cb174a53325dc784c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.895242 13561 tablet_service.cc:1467] Processing CreateTablet for tablet 9e18282dc36248139609c3ad062a0e6d (DEFAULT_TABLE table=rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.895581 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e18282dc36248139609c3ad062a0e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.895745 15776 tablet_bootstrap.cc:654] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.897809 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 718223dd8fd3432cb174a53325dc784c (DEFAULT_TABLE table=rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.897826 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 718223dd8fd3432cb174a53325dc784c (DEFAULT_TABLE table=rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.897980 13428 tablet_service.cc:1467] Processing CreateTablet for tablet 9e18282dc36248139609c3ad062a0e6d (DEFAULT_TABLE table=rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.898159 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718223dd8fd3432cb174a53325dc784c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.898180 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718223dd8fd3432cb174a53325dc784c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.898496 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e18282dc36248139609c3ad062a0e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.914520 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 9e18282dc36248139609c3ad062a0e6d (DEFAULT_TABLE table=rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:45.916911 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e18282dc36248139609c3ad062a0e6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:45.919878 15780 tablet_bootstrap.cc:492] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:45.920025 15777 tablet_bootstrap.cc:492] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:45.922420 15780 tablet_bootstrap.cc:654] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.923049 15777 tablet_bootstrap.cc:654] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.931142 15779 tablet_bootstrap.cc:492] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:45.931841 15780 tablet_bootstrap.cc:492] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:45.932065 15777 tablet_bootstrap.cc:492] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:45.932123 15780 ts_tablet_manager.cc:1367] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 14:14:45.932252 15777 ts_tablet_manager.cc:1367] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240502 14:14:45.932763 15779 tablet_bootstrap.cc:654] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.933086 15777 raft_consensus.cc:348] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:45.933108 15780 raft_consensus.cc:348] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:45.933379 15777 raft_consensus.cc:374] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.933460 15780 raft_consensus.cc:374] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.933506 15777 raft_consensus.cc:729] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.933792 15780 raft_consensus.cc:729] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.933831 15777 consensus_queue.cc:260] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:45.934337 15780 consensus_queue.cc:260] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:45.934554 15777 ts_tablet_manager.cc:1398] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:45.934820 15780 ts_tablet_manager.cc:1398] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:45.934825 15777 tablet_bootstrap.cc:492] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:45.936072 15777 tablet_bootstrap.cc:654] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.936972 15776 tablet_bootstrap.cc:492] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:45.937211 15776 ts_tablet_manager.cc:1367] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.017s
I20240502 14:14:45.937700 15777 tablet_bootstrap.cc:492] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:45.937847 15777 ts_tablet_manager.cc:1367] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:45.938437 15777 raft_consensus.cc:348] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:45.938606 15777 raft_consensus.cc:374] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.938683 15777 raft_consensus.cc:729] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.938901 15777 consensus_queue.cc:260] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:45.939174 15776 raft_consensus.cc:348] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:45.939348 15777 ts_tablet_manager.cc:1398] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:14:45.939381 15776 raft_consensus.cc:374] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.939533 15776 raft_consensus.cc:729] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.939616 15777 tablet_bootstrap.cc:492] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:45.939752 15776 consensus_queue.cc:260] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:45.940411 15776 ts_tablet_manager.cc:1398] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:45.941183 15777 tablet_bootstrap.cc:654] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.941526 15779 tablet_bootstrap.cc:492] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:45.941648 15779 ts_tablet_manager.cc:1367] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240502 14:14:45.942296 15779 raft_consensus.cc:348] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:45.942466 15779 raft_consensus.cc:374] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.942541 15779 raft_consensus.cc:729] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.942721 15779 consensus_queue.cc:260] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:45.943190 15779 ts_tablet_manager.cc:1398] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:14:45.943452 15779 tablet_bootstrap.cc:492] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:45.944079 15776 tablet_bootstrap.cc:492] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:45.944494 15777 tablet_bootstrap.cc:492] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:45.944597 15777 ts_tablet_manager.cc:1367] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:14:45.944810 15779 tablet_bootstrap.cc:654] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.945264 15777 raft_consensus.cc:348] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:45.945412 15777 raft_consensus.cc:374] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.945489 15777 raft_consensus.cc:729] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.945529 15776 tablet_bootstrap.cc:654] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.945662 15777 consensus_queue.cc:260] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:45.947300 15777 ts_tablet_manager.cc:1398] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:45.948429 15776 tablet_bootstrap.cc:492] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:45.948612 15776 ts_tablet_manager.cc:1367] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240502 14:14:45.949440 15776 raft_consensus.cc:348] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:45.949654 15776 raft_consensus.cc:374] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.949741 15776 raft_consensus.cc:729] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.949968 15776 consensus_queue.cc:260] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:45.950564 15776 ts_tablet_manager.cc:1398] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:45.950829 15776 tablet_bootstrap.cc:492] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:45.952001 15776 tablet_bootstrap.cc:654] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:45.953739 15776 tablet_bootstrap.cc:492] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:45.953904 15776 ts_tablet_manager.cc:1367] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.001s
I20240502 14:14:45.954550 15776 raft_consensus.cc:348] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:45.954682 15776 raft_consensus.cc:374] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.954802 15776 raft_consensus.cc:729] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.955040 15776 consensus_queue.cc:260] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:45.955528 15776 ts_tablet_manager.cc:1398] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:14:45.957597 15779 tablet_bootstrap.cc:492] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:45.957777 15779 ts_tablet_manager.cc:1367] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20240502 14:14:45.958496 15779 raft_consensus.cc:348] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:45.958724 15779 raft_consensus.cc:374] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:45.958808 15779 raft_consensus.cc:729] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:45.959059 15779 consensus_queue.cc:260] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:45.959575 15779 ts_tablet_manager.cc:1398] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:45.974654 15666 raft_consensus.cc:482] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:45.974795 15666 raft_consensus.cc:504] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:45.975350 15666 leader_election.cc:288] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:45.975616 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b57323bd14448b9406ce53ada769e5" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:45.975713 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b57323bd14448b9406ce53ada769e5" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:45.975790 13317 raft_consensus.cc:2454] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:45.975886 13188 raft_consensus.cc:2454] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:45.976303 13516 leader_election.cc:302] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:45.976558 15666 raft_consensus.cc:2781] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:45.976657 15666 raft_consensus.cc:482] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:45.976739 15666 raft_consensus.cc:3037] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:45.978551 15666 raft_consensus.cc:504] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:45.979046 15666 leader_election.cc:288] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:45.979332 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b57323bd14448b9406ce53ada769e5" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:45.979521 13317 raft_consensus.cc:3037] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:45.979506 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3b57323bd14448b9406ce53ada769e5" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:45.979652 13188 raft_consensus.cc:3037] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:45.982057 13317 raft_consensus.cc:2454] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:45.982076 13188 raft_consensus.cc:2454] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:45.982484 13516 leader_election.cc:302] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:45.982831 15666 raft_consensus.cc:2781] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:45.982961 15666 raft_consensus.cc:686] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:45.983186 15666 consensus_queue.cc:237] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:45.986404 13042 catalog_manager.cc:5484] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:46.019976 15768 raft_consensus.cc:482] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:46.020098 15768 raft_consensus.cc:504] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.020722 15768 leader_election.cc:288] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:46.020983 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718223dd8fd3432cb174a53325dc784c" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:46.021095 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718223dd8fd3432cb174a53325dc784c" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:46.021168 13581 raft_consensus.cc:2454] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:46.021299 13188 raft_consensus.cc:2454] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:46.021566 13251 leader_election.cc:302] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:46.021783 15768 raft_consensus.cc:2781] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:46.021879 15768 raft_consensus.cc:482] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:46.021953 15768 raft_consensus.cc:3037] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:46.023466 15768 raft_consensus.cc:504] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.023891 15768 leader_election.cc:288] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:46.024255 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718223dd8fd3432cb174a53325dc784c" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:46.024264 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "718223dd8fd3432cb174a53325dc784c" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:46.024423 13581 raft_consensus.cc:3037] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:46.024444 13188 raft_consensus.cc:3037] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:46.027015 13581 raft_consensus.cc:2454] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:46.027005 13188 raft_consensus.cc:2454] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:46.027364 13251 leader_election.cc:302] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:46.027576 15768 raft_consensus.cc:2781] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:46.027750 15768 raft_consensus.cc:686] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:46.028074 15768 consensus_queue.cc:237] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.031212 13042 catalog_manager.cc:5484] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:46.068938 15666 raft_consensus.cc:482] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:46.069095 15666 raft_consensus.cc:504] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:46.069635 15666 leader_election.cc:288] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:46.069970 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e18282dc36248139609c3ad062a0e6d" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:46.070185 13317 raft_consensus.cc:2454] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:46.070127 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e18282dc36248139609c3ad062a0e6d" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:46.070333 13447 raft_consensus.cc:2454] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:46.070524 13516 leader_election.cc:302] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:46.070787 15666 raft_consensus.cc:2781] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:46.070858 15666 raft_consensus.cc:482] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:46.070955 15666 raft_consensus.cc:3037] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:46.072584 15666 raft_consensus.cc:504] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:46.073127 15666 leader_election.cc:288] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:46.073450 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e18282dc36248139609c3ad062a0e6d" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:46.073611 13317 raft_consensus.cc:3037] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:46.073556 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e18282dc36248139609c3ad062a0e6d" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:46.073740 13447 raft_consensus.cc:3037] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:46.075418 13317 raft_consensus.cc:2454] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:46.075735 13447 raft_consensus.cc:2454] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:46.075742 13516 leader_election.cc:302] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:46.076000 15666 raft_consensus.cc:2781] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:46.076157 15666 raft_consensus.cc:686] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:46.076391 15666 consensus_queue.cc:237] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:14:46.079401 13042 catalog_manager.cc:5484] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:46.360800 15666 consensus_queue.cc:1035] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:46.375123 15770 consensus_queue.cc:1035] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:46.437844 15792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:46.438194 15792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:46.444572 15792 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_18'
I20240502 14:14:46.545413 15768 consensus_queue.cc:1035] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:46.550490 15677 consensus_queue.cc:1035] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:46.575767 15666 consensus_queue.cc:1035] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:46.661371 13516 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.194:33193, user_credentials={real_user=slave}} blocked reactor thread for 44069.5us
I20240502 14:14:46.691845 15666 consensus_queue.cc:1035] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 295.251 ms
time per row: 0.147626 ms
I20240502 14:14:46.879577 13045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:46.889007 13428 tablet_service.cc:1467] Processing CreateTablet for tablet 7852539de07744caaf385620c5166861 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:46.889482 13428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7852539de07744caaf385620c5166861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:46.889925 13559 tablet_service.cc:1467] Processing CreateTablet for tablet db06293ca0914bb88dbff3808e3f1874 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:46.890288 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db06293ca0914bb88dbff3808e3f1874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:46.891184 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 0854e9f4851e400f9af18181a71ae8f7 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:46.891535 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0854e9f4851e400f9af18181a71ae8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:46.893604 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 7852539de07744caaf385620c5166861 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:46.893761 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 0854e9f4851e400f9af18181a71ae8f7 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:46.893957 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7852539de07744caaf385620c5166861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:46.894079 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0854e9f4851e400f9af18181a71ae8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:46.897521 13429 tablet_service.cc:1467] Processing CreateTablet for tablet db06293ca0914bb88dbff3808e3f1874 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:46.897563 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 7852539de07744caaf385620c5166861 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:46.897809 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db06293ca0914bb88dbff3808e3f1874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:46.897931 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7852539de07744caaf385620c5166861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:46.900812 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 0854e9f4851e400f9af18181a71ae8f7 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:46.901155 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0854e9f4851e400f9af18181a71ae8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:46.902689 15864 tablet_bootstrap.cc:492] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:46.904104 15864 tablet_bootstrap.cc:654] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:46.905778 15865 tablet_bootstrap.cc:492] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:46.908402 13166 tablet_service.cc:1467] Processing CreateTablet for tablet db06293ca0914bb88dbff3808e3f1874 (DEFAULT_TABLE table=rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:46.908948 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db06293ca0914bb88dbff3808e3f1874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:46.914494 15867 tablet_bootstrap.cc:492] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:46.919308 15865 tablet_bootstrap.cc:654] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:46.919958 15867 tablet_bootstrap.cc:654] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:46.921864 15868 tablet_bootstrap.cc:492] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:46.922106 15864 tablet_bootstrap.cc:492] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:46.922227 15864 ts_tablet_manager.cc:1367] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.018s
I20240502 14:14:46.922748 15864 raft_consensus.cc:348] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:46.922896 15864 raft_consensus.cc:374] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:46.922976 15864 raft_consensus.cc:729] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:46.923175 15864 consensus_queue.cc:260] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:46.923789 15864 ts_tablet_manager.cc:1398] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:14:46.924295 15868 tablet_bootstrap.cc:654] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:46.925141 15865 tablet_bootstrap.cc:492] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:46.925644 15865 ts_tablet_manager.cc:1367] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.000s
I20240502 14:14:46.926553 15865 raft_consensus.cc:348] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.926786 15865 raft_consensus.cc:374] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:46.926882 15865 raft_consensus.cc:729] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:46.927207 15865 consensus_queue.cc:260] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.929301 15865 ts_tablet_manager.cc:1398] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:46.929517 15868 tablet_bootstrap.cc:492] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:46.929579 15865 tablet_bootstrap.cc:492] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:46.929726 15868 ts_tablet_manager.cc:1367] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240502 14:14:46.930456 15868 raft_consensus.cc:348] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:46.930675 15868 raft_consensus.cc:374] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:46.930811 15868 raft_consensus.cc:729] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:46.930851 15865 tablet_bootstrap.cc:654] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:46.931072 15868 consensus_queue.cc:260] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:46.931627 15868 ts_tablet_manager.cc:1398] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:46.931918 15868 tablet_bootstrap.cc:492] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:46.933485 15868 tablet_bootstrap.cc:654] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:46.933535 15867 tablet_bootstrap.cc:492] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:46.935825 15867 ts_tablet_manager.cc:1367] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20240502 14:14:46.936560 15867 raft_consensus.cc:348] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.936789 15867 raft_consensus.cc:374] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:46.936949 15867 raft_consensus.cc:729] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:46.937263 15867 consensus_queue.cc:260] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.939066 15867 ts_tablet_manager.cc:1398] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:46.939352 15867 tablet_bootstrap.cc:492] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:46.940030 15865 tablet_bootstrap.cc:492] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:46.940299 15865 ts_tablet_manager.cc:1367] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240502 14:14:46.941200 15865 raft_consensus.cc:348] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.941390 15865 raft_consensus.cc:374] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:46.941465 15865 raft_consensus.cc:729] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:46.941761 15864 tablet_bootstrap.cc:492] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:46.941696 15865 consensus_queue.cc:260] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.941803 15868 tablet_bootstrap.cc:492] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:46.941974 15868 ts_tablet_manager.cc:1367] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20240502 14:14:46.943140 15865 ts_tablet_manager.cc:1398] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:46.943161 15864 tablet_bootstrap.cc:654] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:46.943452 15865 tablet_bootstrap.cc:492] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:46.943889 15867 tablet_bootstrap.cc:654] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:46.942718 15868 raft_consensus.cc:348] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.944564 15865 tablet_bootstrap.cc:654] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:46.944605 15868 raft_consensus.cc:374] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:46.944717 15868 raft_consensus.cc:729] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:46.945024 15868 consensus_queue.cc:260] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.945657 15868 ts_tablet_manager.cc:1398] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:14:46.946691 15867 tablet_bootstrap.cc:492] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:46.947012 15867 ts_tablet_manager.cc:1367] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20240502 14:14:46.948272 15867 raft_consensus.cc:348] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.948611 15867 raft_consensus.cc:374] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:46.948686 15867 raft_consensus.cc:729] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:46.949093 15867 consensus_queue.cc:260] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.949610 15867 ts_tablet_manager.cc:1398] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:46.951083 15865 tablet_bootstrap.cc:492] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:46.951144 15864 tablet_bootstrap.cc:492] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:46.951264 15865 ts_tablet_manager.cc:1367] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.002s
I20240502 14:14:46.951364 15864 ts_tablet_manager.cc:1367] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240502 14:14:46.951930 15865 raft_consensus.cc:348] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:46.952438 15865 raft_consensus.cc:374] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:46.952361 15864 raft_consensus.cc:348] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.952571 15864 raft_consensus.cc:374] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:46.952600 15865 raft_consensus.cc:729] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:46.952780 15864 raft_consensus.cc:729] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:46.953042 15864 consensus_queue.cc:260] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.953272 15865 consensus_queue.cc:260] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:46.953748 15864 ts_tablet_manager.cc:1398] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:46.954986 15865 ts_tablet_manager.cc:1398] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:14:46.963258 15666 raft_consensus.cc:482] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:46.963404 15666 raft_consensus.cc:504] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.964057 15666 leader_election.cc:288] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:46.964494 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db06293ca0914bb88dbff3808e3f1874" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:46.964694 13445 raft_consensus.cc:2454] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:46.964663 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db06293ca0914bb88dbff3808e3f1874" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:46.964883 13188 raft_consensus.cc:2454] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:46.965035 13514 leader_election.cc:302] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:46.965354 15841 raft_consensus.cc:2781] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:46.965453 15841 raft_consensus.cc:482] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:46.965572 15841 raft_consensus.cc:3037] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:46.967761 15841 raft_consensus.cc:504] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.968286 15841 leader_election.cc:288] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:46.968603 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db06293ca0914bb88dbff3808e3f1874" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:46.968676 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db06293ca0914bb88dbff3808e3f1874" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:46.968789 13445 raft_consensus.cc:3037] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:46.968842 13188 raft_consensus.cc:3037] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:46.971405 13188 raft_consensus.cc:2454] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:46.971406 13445 raft_consensus.cc:2454] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:46.971720 13514 leader_election.cc:302] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:46.971947 15841 raft_consensus.cc:2781] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:46.972074 15841 raft_consensus.cc:686] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:46.972271 15841 consensus_queue.cc:237] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:46.974596 13045 catalog_manager.cc:5484] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:46.999456 15768 raft_consensus.cc:482] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:46.999584 15768 raft_consensus.cc:504] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:47.000146 15768 leader_election.cc:288] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:47.000456 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0854e9f4851e400f9af18181a71ae8f7" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:47.000520 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0854e9f4851e400f9af18181a71ae8f7" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:47.000633 13579 raft_consensus.cc:2454] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:47.000705 13445 raft_consensus.cc:2454] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:47.000996 13251 leader_election.cc:302] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:47.001309 15768 raft_consensus.cc:2781] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:47.001439 15768 raft_consensus.cc:482] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:47.001549 15768 raft_consensus.cc:3037] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:47.003394 15768 raft_consensus.cc:504] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:47.003962 15768 leader_election.cc:288] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:47.004226 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0854e9f4851e400f9af18181a71ae8f7" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:47.004354 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0854e9f4851e400f9af18181a71ae8f7" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:47.004400 13579 raft_consensus.cc:3037] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:47.004536 13445 raft_consensus.cc:3037] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:47.006800 13445 raft_consensus.cc:2454] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:47.006788 13579 raft_consensus.cc:2454] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:47.007133 13251 leader_election.cc:302] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:47.007443 15768 raft_consensus.cc:2781] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:47.007596 15768 raft_consensus.cc:686] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:47.007844 15768 consensus_queue.cc:237] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:47.010677 13045 catalog_manager.cc:5484] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:47.146728 15771 raft_consensus.cc:482] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:47.146915 15771 raft_consensus.cc:504] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:47.147573 15771 leader_election.cc:288] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:47.147895 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852539de07744caaf385620c5166861" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:47.148154 13317 raft_consensus.cc:2454] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:47.148360 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852539de07744caaf385620c5166861" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:47.148569 13384 leader_election.cc:302] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:47.148613 13188 raft_consensus.cc:2454] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:47.148844 15771 raft_consensus.cc:2781] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:47.148994 15771 raft_consensus.cc:482] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:47.149149 15771 raft_consensus.cc:3037] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:47.151149 15771 raft_consensus.cc:504] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:47.151556 15771 leader_election.cc:288] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:47.151888 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852539de07744caaf385620c5166861" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:47.151947 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852539de07744caaf385620c5166861" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:47.152063 13317 raft_consensus.cc:3037] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:47.152104 13188 raft_consensus.cc:3037] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:47.153806 13317 raft_consensus.cc:2454] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:47.153831 13188 raft_consensus.cc:2454] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:47.154183 13384 leader_election.cc:302] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:47.154454 15771 raft_consensus.cc:2781] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:47.154557 15771 raft_consensus.cc:686] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:47.154793 15771 consensus_queue.cc:237] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:47.158030 13045 catalog_manager.cc:5484] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:47.381760 15841 consensus_queue.cc:1035] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:47.395753 15839 consensus_queue.cc:1035] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:47.540378 15880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:47.540633 15880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:47.547005 15880 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240502 14:14:47.583724 15853 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:47.589804 15853 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:47.621400 15609 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:47.627708 15885 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 232.413 ms
time per row: 0.116207 ms
I20240502 14:14:47.934466 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.197:43695
--local_ip_for_outbound_sockets=127.6.196.197
--tserver_master_addrs=127.6.196.254:44303
--webserver_port=34095
--webserver_interface=127.6.196.197
--builtin_ntp_servers=127.6.196.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:14:48.253065 15940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:48.253378 15940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:48.253482 15940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:48.260821 15940 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:48.260915 15940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:48.261008 15940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.197
I20240502 14:14:48.268815 15940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.197:43695
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.196.197
--webserver_port=34095
--tserver_master_addrs=127.6.196.254:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.197
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:48.270221 15940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:48.271847 15940 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:48.287371 15945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:48.288166 15946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:48.288615 15948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:48.289693 15940 server_base.cc:1034] running on GCE node
I20240502 14:14:48.290191 15940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:48.290997 15940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:48.292201 15940 hybrid_clock.cc:648] HybridClock initialized: now 1714659288291964 us; error 224 us; skew 500 ppm
I20240502 14:14:48.292536 15940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:48.295111 15940 webserver.cc:460] Webserver started at http://127.6.196.197:34095/ using document root <none> and password file <none>
I20240502 14:14:48.295680 15940 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:48.295789 15940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:48.300794 15940 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:14:48.303828 15954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:48.305073 15940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:48.305250 15940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "d1da7f11f0604f78934354a992bb0f55"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-kjf7"
I20240502 14:14:48.306061 15940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:48.346735 15940 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:48.347461 15940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:48.347728 15940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:48.348567 15940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:48.349973 15940 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:48.350046 15940 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:48.350183 15940 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:48.350239 15940 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:48.396492 15940 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.197:43695
I20240502 14:14:48.396540 16067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.197:43695 every 8 connection(s)
I20240502 14:14:48.397818 15940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:14:48.403174 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 15940
I20240502 14:14:48.403689 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.198:42041
--local_ip_for_outbound_sockets=127.6.196.198
--tserver_master_addrs=127.6.196.254:44303
--webserver_port=33425
--webserver_interface=127.6.196.198
--builtin_ntp_servers=127.6.196.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:48.410749 16068 heartbeater.cc:347] Connected to a master server at 127.6.196.254:44303
I20240502 14:14:48.410990 16068 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:48.411834 16068 heartbeater.cc:511] Master 127.6.196.254:44303 requested a full tablet report, sending...
I20240502 14:14:48.413249 13042 ts_manager.cc:194] Re-registered known tserver with Master: d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:48.413954 13042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.197:51629
W20240502 14:14:48.724481 16072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:48.724752 16072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:48.724886 16072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:48.732453 16072 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:48.732579 16072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:48.732709 16072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.198
I20240502 14:14:48.742378 16072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.198:42041
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.196.198
--webserver_port=33425
--tserver_master_addrs=127.6.196.254:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.198
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:48.743551 16072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:48.745282 16072 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:48.756476 16078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:48.757017 16080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:48.757634 16077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:48.758509 16072 server_base.cc:1034] running on GCE node
I20240502 14:14:48.759029 16072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:48.759877 16072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:48.761086 16072 hybrid_clock.cc:648] HybridClock initialized: now 1714659288761051 us; error 39 us; skew 500 ppm
I20240502 14:14:48.761417 16072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:48.768894 16072 webserver.cc:460] Webserver started at http://127.6.196.198:33425/ using document root <none> and password file <none>
I20240502 14:14:48.769624 16072 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:48.769735 16072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:48.774240 16072 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20240502 14:14:48.776928 16087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:48.778204 16072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:14:48.778347 16072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "c86f7440570f4796babfd44c3d7fd847"
format_stamp: "Formatted at 2024-05-02 14:14:14 on dist-test-slave-kjf7"
I20240502 14:14:48.779100 16072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:48.801470 16072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:48.802080 16072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:48.802281 16072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:48.802949 16072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:48.804152 16072 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:48.804217 16072 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:48.804327 16072 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:48.804373 16072 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:48.837055 16072 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.198:42041
I20240502 14:14:48.837116 16199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.198:42041 every 8 connection(s)
I20240502 14:14:48.838413 16072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:14:48.844666 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 16072
I20240502 14:14:48.845161 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.199:33857
--local_ip_for_outbound_sockets=127.6.196.199
--tserver_master_addrs=127.6.196.254:44303
--webserver_port=35701
--webserver_interface=127.6.196.199
--builtin_ntp_servers=127.6.196.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:48.850281 16200 heartbeater.cc:347] Connected to a master server at 127.6.196.254:44303
I20240502 14:14:48.850518 16200 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:48.851258 16200 heartbeater.cc:511] Master 127.6.196.254:44303 requested a full tablet report, sending...
I20240502 14:14:48.852440 13042 ts_manager.cc:194] Re-registered known tserver with Master: c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:48.853094 13042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.198:54515
W20240502 14:14:49.161562 16203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:49.161820 16203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:49.161914 16203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:49.168722 16203 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:49.168830 16203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:49.168941 16203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.199
I20240502 14:14:49.177155 16203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.199:33857
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.196.199
--webserver_port=35701
--tserver_master_addrs=127.6.196.254:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.199
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:49.178193 16203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:49.179687 16203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:49.190544 16209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:49.191345 16210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:49.191828 16212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:49.192456 16203 server_base.cc:1034] running on GCE node
I20240502 14:14:49.192976 16203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:49.193919 16203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:49.195156 16203 hybrid_clock.cc:648] HybridClock initialized: now 1714659289195115 us; error 31 us; skew 500 ppm
I20240502 14:14:49.195437 16203 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:49.197340 16203 webserver.cc:460] Webserver started at http://127.6.196.199:35701/ using document root <none> and password file <none>
I20240502 14:14:49.197925 16203 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:49.198047 16203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:49.202878 16203 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 14:14:49.205444 16218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:49.206445 16203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240502 14:14:49.206576 16203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "5297e816f782430d9a01220f0471a882"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-kjf7"
I20240502 14:14:49.207312 16203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:49.233644 16203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:49.234324 16203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:49.234544 16203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:49.245203 16203 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:49.246687 16203 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:49.246755 16203 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:49.246873 16203 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:49.246933 16203 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:49.297345 16203 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.199:33857
I20240502 14:14:49.297406 16332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.199:33857 every 8 connection(s)
I20240502 14:14:49.299023 16203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:14:49.305871 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 16203
I20240502 14:14:49.306602 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.200:43857
--local_ip_for_outbound_sockets=127.6.196.200
--tserver_master_addrs=127.6.196.254:44303
--webserver_port=44151
--webserver_interface=127.6.196.200
--builtin_ntp_servers=127.6.196.212:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:14:49.311084 16333 heartbeater.cc:347] Connected to a master server at 127.6.196.254:44303
I20240502 14:14:49.311362 16333 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:49.312233 16333 heartbeater.cc:511] Master 127.6.196.254:44303 requested a full tablet report, sending...
I20240502 14:14:49.313609 13042 ts_manager.cc:194] Re-registered known tserver with Master: 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:14:49.314630 13042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.199:36841
I20240502 14:14:49.415920 16068 heartbeater.cc:503] Master 127.6.196.254:44303 was elected leader, sending a full tablet report...
W20240502 14:14:49.652172 16336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:49.652446 16336 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:14:49.652560 16336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:49.659724 16336 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:14:49.659814 16336 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:14:49.659938 16336 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.200
I20240502 14:14:49.668581 16336 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.200:43857
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.196.200
--webserver_port=44151
--tserver_master_addrs=127.6.196.254:44303
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.200
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:14:49.670094 16336 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:14:49.671705 16336 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:14:49.685703 16344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:49.686169 16343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:14:49.686950 16346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:14:49.687801 16336 server_base.cc:1034] running on GCE node
I20240502 14:14:49.688374 16336 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:14:49.689256 16336 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:14:49.690459 16336 hybrid_clock.cc:648] HybridClock initialized: now 1714659289690422 us; error 33 us; skew 500 ppm
I20240502 14:14:49.690726 16336 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:14:49.692662 16336 webserver.cc:460] Webserver started at http://127.6.196.200:44151/ using document root <none> and password file <none>
I20240502 14:14:49.693282 16336 fs_manager.cc:359] Metadata directory not provided
I20240502 14:14:49.693398 16336 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:14:49.698437 16336 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:14:49.700927 16352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:49.702328 16336 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:14:49.702488 16336 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "8b47d4390f8343a493d6c00123785968"
format_stamp: "Formatted at 2024-05-02 14:14:15 on dist-test-slave-kjf7"
I20240502 14:14:49.703346 16336 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:14:49.717483 16336 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:14:49.718089 16336 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:14:49.718348 16336 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:14:49.719218 16336 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:14:49.720757 16336 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:14:49.720834 16336 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:49.720938 16336 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:14:49.720988 16336 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:14:49.762730 16336 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.200:43857
I20240502 14:14:49.762782 16466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.200:43857 every 8 connection(s)
I20240502 14:14:49.764005 16336 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:14:49.766952 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 16336
I20240502 14:14:49.777760 16467 heartbeater.cc:347] Connected to a master server at 127.6.196.254:44303
I20240502 14:14:49.778007 16467 heartbeater.cc:464] Registering TS with master...
I20240502 14:14:49.778753 16467 heartbeater.cc:511] Master 127.6.196.254:44303 requested a full tablet report, sending...
I20240502 14:14:49.780117 13042 ts_manager.cc:194] Re-registered known tserver with Master: 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:49.781020 13042 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.200:56627
I20240502 14:14:49.782897 13042 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:49.794827 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 67e2fe12c01e4cf7994eed72f0fa6776 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5d9a14d0d9de4fcdb5f09d781e512363]), partition=
I20240502 14:14:49.795344 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e2fe12c01e4cf7994eed72f0fa6776. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.808843 16401 tablet_service.cc:1467] Processing CreateTablet for tablet 67e2fe12c01e4cf7994eed72f0fa6776 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5d9a14d0d9de4fcdb5f09d781e512363]), partition=
I20240502 14:14:49.809952 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e2fe12c01e4cf7994eed72f0fa6776. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.811702 16483 tablet_bootstrap.cc:492] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:49.813345 16483 tablet_bootstrap.cc:654] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.815598 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 67e2fe12c01e4cf7994eed72f0fa6776 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5d9a14d0d9de4fcdb5f09d781e512363]), partition=
I20240502 14:14:49.816414 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67e2fe12c01e4cf7994eed72f0fa6776. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:49.826951 16483 tablet_bootstrap.cc:492] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:49.827148 16483 ts_tablet_manager.cc:1367] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20240502 14:14:49.827965 16483 raft_consensus.cc:348] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:49.829453 16483 raft_consensus.cc:374] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.829576 16483 raft_consensus.cc:729] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.829840 16483 consensus_queue.cc:260] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:49.834797 16483 ts_tablet_manager.cc:1398] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:14:49.840550 16485 tablet_bootstrap.cc:492] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:49.843590 16485 tablet_bootstrap.cc:654] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.844818 16485 log.cc:824] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:49.848134 16486 tablet_bootstrap.cc:492] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:49.850252 16485 tablet_bootstrap.cc:492] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:49.851519 16485 ts_tablet_manager.cc:1367] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240502 14:14:49.852360 16486 tablet_bootstrap.cc:654] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:49.853456 16486 log.cc:824] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:49.855417 16200 heartbeater.cc:503] Master 127.6.196.254:44303 was elected leader, sending a full tablet report...
I20240502 14:14:49.862820 16485 raft_consensus.cc:348] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:49.863193 16485 raft_consensus.cc:374] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.863288 16485 raft_consensus.cc:729] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.863989 16485 consensus_queue.cc:260] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:49.865536 16485 ts_tablet_manager.cc:1398] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.014s user 0.008s sys 0.003s
I20240502 14:14:49.866602 16467 heartbeater.cc:503] Master 127.6.196.254:44303 was elected leader, sending a full tablet report...
I20240502 14:14:49.879330 16486 tablet_bootstrap.cc:492] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:49.879637 16486 ts_tablet_manager.cc:1367] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.004s
I20240502 14:14:49.887832 16486 raft_consensus.cc:348] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:49.888283 16486 raft_consensus.cc:374] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:49.888401 16486 raft_consensus.cc:729] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:49.889062 16486 consensus_queue.cc:260] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:49.890515 16486 ts_tablet_manager.cc:1398] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20240502 14:14:49.982653 16489 raft_consensus.cc:482] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:49.983054 16489 raft_consensus.cc:504] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:49.984987 16489 leader_election.cc:288] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:49.994307 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e2fe12c01e4cf7994eed72f0fa6776" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:49.994730 16155 raft_consensus.cc:2454] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:14:49.995368 16354 leader_election.cc:302] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:49.995780 16489 raft_consensus.cc:2781] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:49.995914 16489 raft_consensus.cc:482] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:49.996011 16489 raft_consensus.cc:3037] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:49.998438 16489 raft_consensus.cc:504] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:49.999159 16489 leader_election.cc:288] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:49.999840 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e2fe12c01e4cf7994eed72f0fa6776" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:50.000095 16155 raft_consensus.cc:3037] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.003147 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e2fe12c01e4cf7994eed72f0fa6776" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:50.003222 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67e2fe12c01e4cf7994eed72f0fa6776" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:50.003262 16155 raft_consensus.cc:2454] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:14:50.003391 13187 raft_consensus.cc:2454] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:14:50.003780 16354 leader_election.cc:302] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:50.004083 16489 raft_consensus.cc:2781] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:50.005489 16489 raft_consensus.cc:686] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:14:50.005939 16489 consensus_queue.cc:237] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:50.010341 13043 catalog_manager.cc:5484] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:50.015496 16468 tablet.cc:2369] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:14:50.094980 16201 tablet.cc:2369] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:50.132489 13043 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:50.142360 13188 raft_consensus.cc:3037] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.142988 16155 raft_consensus.cc:1261] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:50.144359 16493 consensus_queue.cc:1035] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.145080 13188 raft_consensus.cc:1261] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:50.145854 16493 consensus_queue.cc:1035] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.165012 16500 tablet.cc:1722] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:50.165984 16501 tablet.cc:1722] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:50.166381 16500 tablet.cc:1620] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:14:50.167203 16501 tablet.cc:1620] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:14:50.172111 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=5d9a14d0d9de4fcdb5f09d781e512363] alter complete (version 1)
I20240502 14:14:50.176620 13043 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet 67e2fe12c01e4cf7994eed72f0fa6776 (table rebalancer_dynamic_table [id=5d9a14d0d9de4fcdb5f09d781e512363]). Expected version 1 got 0
I20240502 14:14:50.178156 13042 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 67e2fe12c01e4cf7994eed72f0fa6776 (table rebalancer_dynamic_table [id=5d9a14d0d9de4fcdb5f09d781e512363]). Expected version 1 got 0
I20240502 14:14:50.193707 16506 tablet.cc:1722] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:50.194952 16506 tablet.cc:1620] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:50.195086 13042 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 67e2fe12c01e4cf7994eed72f0fa6776 (table rebalancer_dynamic_table [id=5d9a14d0d9de4fcdb5f09d781e512363]). Expected version 1 got 0
I20240502 14:14:50.269248 13042 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:50.275051 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:50.275481 13043 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 14:14:50.275664 13043 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240502 14:14:50.287016 16500 tablet.cc:1722] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:50.287448 16500 tablet.cc:1620] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:14:50.287433 13559 tablet_service.cc:1467] Processing CreateTablet for tablet eab73d39b1f7455bba7aef5e515805c7 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=a6584ce357f343ffa6c92e7412b11020]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.288031 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eab73d39b1f7455bba7aef5e515805c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.288439 16506 tablet.cc:1722] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:50.288858 16506 tablet.cc:1620] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:50.297766 13298 tablet_service.cc:1467] Processing CreateTablet for tablet eab73d39b1f7455bba7aef5e515805c7 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=a6584ce357f343ffa6c92e7412b11020]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.298337 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eab73d39b1f7455bba7aef5e515805c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.302078 16135 tablet_service.cc:1467] Processing CreateTablet for tablet f64bbd1ae9074071825459e558f03ff5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=a6584ce357f343ffa6c92e7412b11020]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.302587 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64bbd1ae9074071825459e558f03ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.309405 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 536c22c444344bb485d3e522f0f77154 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=a6584ce357f343ffa6c92e7412b11020]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.309509 13043 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet fd031f1ac25b42769006740ba81a96b9
I20240502 14:14:50.309967 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536c22c444344bb485d3e522f0f77154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.310364 13043 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 257d72583cd94c0d8efdd8925be48dca
I20240502 14:14:50.310868 13043 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet a5969155f010461293a7f2f61c88c817
I20240502 14:14:50.312381 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 257d72583cd94c0d8efdd8925be48dca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:50.312506 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet fd031f1ac25b42769006740ba81a96b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:50.312849 13167 tablet_service.cc:1514] Processing DeleteTablet for tablet a5969155f010461293a7f2f61c88c817 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:50.312973 16501 tablet.cc:1722] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:50.313486 16501 tablet.cc:1620] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:14:50.313493 13429 tablet_service.cc:1514] Processing DeleteTablet for tablet 257d72583cd94c0d8efdd8925be48dca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:50.313928 13428 tablet_service.cc:1514] Processing DeleteTablet for tablet a5969155f010461293a7f2f61c88c817 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:50.314078 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet fd031f1ac25b42769006740ba81a96b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:50.314553 13561 tablet_service.cc:1514] Processing DeleteTablet for tablet a5969155f010461293a7f2f61c88c817 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:50.316911 16333 heartbeater.cc:503] Master 127.6.196.254:44303 was elected leader, sending a full tablet report...
I20240502 14:14:50.328971 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=5d9a14d0d9de4fcdb5f09d781e512363] alter complete (version 2)
I20240502 14:14:50.338982 16486 tablet_bootstrap.cc:492] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:50.340963 16486 tablet_bootstrap.cc:654] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.341754 16533 tablet_bootstrap.cc:492] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:50.344512 16533 tablet_bootstrap.cc:654] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.354380 16532 tablet_bootstrap.cc:492] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:50.362541 16532 tablet_bootstrap.cc:654] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.367921 16527 tablet_replica.cc:330] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:50.369537 16529 tablet_replica.cc:330] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:50.369971 16529 raft_consensus.cc:2229] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.370275 16529 raft_consensus.cc:2258] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.370694 16533 tablet_bootstrap.cc:492] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:50.370820 16533 ts_tablet_manager.cc:1367] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.000s
I20240502 14:14:50.371423 16529 ts_tablet_manager.cc:1875] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.371436 16533 raft_consensus.cc:348] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:50.371621 16533 raft_consensus.cc:374] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.371693 16533 raft_consensus.cc:729] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.371888 16533 consensus_queue.cc:260] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:50.372332 16533 ts_tablet_manager.cc:1398] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:14:50.374679 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 536c22c444344bb485d3e522f0f77154 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=a6584ce357f343ffa6c92e7412b11020]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.375105 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536c22c444344bb485d3e522f0f77154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.376806 16525 tablet_bootstrap.cc:492] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:50.378685 16525 tablet_bootstrap.cc:654] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.379674 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet fd031f1ac25b42769006740ba81a96b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:50.379734 16002 tablet_service.cc:1467] Processing CreateTablet for tablet f64bbd1ae9074071825459e558f03ff5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=a6584ce357f343ffa6c92e7412b11020]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.380519 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet 257d72583cd94c0d8efdd8925be48dca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:50.381505 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64bbd1ae9074071825459e558f03ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.382740 16003 tablet_service.cc:1467] Processing CreateTablet for tablet eab73d39b1f7455bba7aef5e515805c7 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=a6584ce357f343ffa6c92e7412b11020]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.383154 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eab73d39b1f7455bba7aef5e515805c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.384418 16532 tablet_bootstrap.cc:492] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:50.385437 16532 ts_tablet_manager.cc:1367] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.003s
I20240502 14:14:50.386549 16532 raft_consensus.cc:348] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:50.386870 16532 raft_consensus.cc:374] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.387085 16538 tablet_replica.cc:330] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:50.387400 16538 raft_consensus.cc:2229] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.388005 16538 raft_consensus.cc:2258] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.388414 16532 raft_consensus.cc:729] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.388706 16532 consensus_queue.cc:260] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:50.387744 16525 tablet_bootstrap.cc:492] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:50.390081 16525 ts_tablet_manager.cc:1367] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20240502 14:14:50.391144 16525 raft_consensus.cc:348] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:50.392230 16532 ts_tablet_manager.cc:1398] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:14:50.393324 16525 raft_consensus.cc:374] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.393429 16525 raft_consensus.cc:729] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.394546 16538 ts_tablet_manager.cc:1875] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.395617 16529 ts_tablet_manager.cc:1888] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.395790 16529 log.cc:1196] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/fd031f1ac25b42769006740ba81a96b9
I20240502 14:14:50.395437 16525 consensus_queue.cc:260] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:50.398602 16525 ts_tablet_manager.cc:1398] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240502 14:14:50.398844 16525 tablet_bootstrap.cc:492] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:50.401790 16486 tablet_bootstrap.cc:492] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:50.401993 16486 ts_tablet_manager.cc:1367] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.005s
I20240502 14:14:50.402052 16525 tablet_bootstrap.cc:654] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.403012 16486 raft_consensus.cc:348] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:50.403193 16486 raft_consensus.cc:374] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.403290 16486 raft_consensus.cc:729] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.404354 16529 ts_tablet_manager.cc:1909] T fd031f1ac25b42769006740ba81a96b9 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:50.405920 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet fd031f1ac25b42769006740ba81a96b9 (table rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]) successfully deleted
I20240502 14:14:50.406317 16529 tablet_replica.cc:330] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:50.406610 16529 raft_consensus.cc:2229] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.407227 16529 raft_consensus.cc:2258] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.408126 16529 ts_tablet_manager.cc:1875] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.408105 16486 consensus_queue.cc:260] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:50.408885 16486 ts_tablet_manager.cc:1398] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:14:50.409722 16539 tablet_bootstrap.cc:492] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:50.368528 16527 raft_consensus.cc:2229] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.411217 16527 raft_consensus.cc:2258] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.415690 16529 ts_tablet_manager.cc:1888] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.416258 16525 tablet_bootstrap.cc:492] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:50.416359 16526 tablet_replica.cc:330] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:50.416641 16529 log.cc:1196] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/a5969155f010461293a7f2f61c88c817
I20240502 14:14:50.416831 16526 raft_consensus.cc:2229] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.417055 16529 ts_tablet_manager.cc:1909] T a5969155f010461293a7f2f61c88c817 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:50.417186 16526 raft_consensus.cc:2258] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.417289 16525 ts_tablet_manager.cc:1367] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20240502 14:14:50.416409 16538 ts_tablet_manager.cc:1888] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.418846 16528 raft_consensus.cc:482] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.418872 16527 ts_tablet_manager.cc:1875] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.418871 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet a5969155f010461293a7f2f61c88c817 (table rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]) successfully deleted
I20240502 14:14:50.419047 16538 log.cc:1196] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/fd031f1ac25b42769006740ba81a96b9
I20240502 14:14:50.418995 16528 raft_consensus.cc:504] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:50.419723 16538 ts_tablet_manager.cc:1909] T fd031f1ac25b42769006740ba81a96b9 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:50.419880 16528 leader_election.cc:288] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:50.420711 16526 ts_tablet_manager.cc:1875] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.420924 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab73d39b1f7455bba7aef5e515805c7" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:50.421159 13579 raft_consensus.cc:2454] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:50.421599 13251 leader_election.cc:302] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:50.421864 16528 raft_consensus.cc:2781] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:50.421968 16528 raft_consensus.cc:482] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:50.422055 16528 raft_consensus.cc:3037] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.422274 16538 tablet_replica.cc:330] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:50.422474 16538 raft_consensus.cc:2229] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.422678 16538 raft_consensus.cc:2258] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.423313 16538 ts_tablet_manager.cc:1875] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.424494 16528 raft_consensus.cc:504] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:50.425030 16528 leader_election.cc:288] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:50.432879 16267 tablet_service.cc:1467] Processing CreateTablet for tablet f64bbd1ae9074071825459e558f03ff5 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=a6584ce357f343ffa6c92e7412b11020]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.434516 16267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64bbd1ae9074071825459e558f03ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.438424 16526 ts_tablet_manager.cc:1888] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.438639 16526 log.cc:1196] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/257d72583cd94c0d8efdd8925be48dca
I20240502 14:14:50.439232 16526 ts_tablet_manager.cc:1909] T 257d72583cd94c0d8efdd8925be48dca P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:50.440091 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab73d39b1f7455bba7aef5e515805c7" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:50.440284 13579 raft_consensus.cc:3037] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.440501 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet fd031f1ac25b42769006740ba81a96b9 (table rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]) successfully deleted
I20240502 14:14:50.441116 16526 tablet_replica.cc:330] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:50.441905 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 536c22c444344bb485d3e522f0f77154 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=a6584ce357f343ffa6c92e7412b11020]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:50.442318 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536c22c444344bb485d3e522f0f77154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.442729 13579 raft_consensus.cc:2454] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:50.439944 16538 ts_tablet_manager.cc:1888] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.443663 16538 log.cc:1196] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/257d72583cd94c0d8efdd8925be48dca
I20240502 14:14:50.444032 16538 ts_tablet_manager.cc:1909] T 257d72583cd94c0d8efdd8925be48dca P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:50.441541 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab73d39b1f7455bba7aef5e515805c7" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:14:50.444873 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 257d72583cd94c0d8efdd8925be48dca (table rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]) successfully deleted
I20240502 14:14:50.445320 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 257d72583cd94c0d8efdd8925be48dca (table rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]) successfully deleted
I20240502 14:14:50.443253 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab73d39b1f7455bba7aef5e515805c7" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
W20240502 14:14:50.446324 13253 leader_election.cc:341] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:14:50.446596 13253 leader_election.cc:341] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:50.447175 13251 leader_election.cc:302] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters: d1da7f11f0604f78934354a992bb0f55
I20240502 14:14:50.447475 16528 raft_consensus.cc:2781] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:50.447557 16526 raft_consensus.cc:2229] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.447729 16528 raft_consensus.cc:686] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:50.447916 16526 raft_consensus.cc:2258] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.447850 16525 raft_consensus.cc:348] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:50.448051 16525 raft_consensus.cc:374] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.448042 16528 consensus_queue.cc:237] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:50.448210 16525 raft_consensus.cc:729] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.448577 16525 consensus_queue.cc:260] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:50.448724 16526 ts_tablet_manager.cc:1875] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.450034 16525 ts_tablet_manager.cc:1398] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.033s user 0.000s sys 0.002s
I20240502 14:14:50.452780 16539 tablet_bootstrap.cc:654] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.454720 16539 log.cc:824] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:50.456199 16526 ts_tablet_manager.cc:1888] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.456377 16526 log.cc:1196] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/a5969155f010461293a7f2f61c88c817
I20240502 14:14:50.456821 16526 ts_tablet_manager.cc:1909] T a5969155f010461293a7f2f61c88c817 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:50.458698 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet a5969155f010461293a7f2f61c88c817 (table rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]) successfully deleted
I20240502 14:14:50.459033 16545 tablet_bootstrap.cc:492] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:50.459136 16526 tablet_replica.cc:330] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:50.459378 16526 raft_consensus.cc:2229] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.459847 16526 raft_consensus.cc:2258] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.460578 16526 ts_tablet_manager.cc:1875] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.463938 16545 tablet_bootstrap.cc:654] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.464789 13041 catalog_manager.cc:5484] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:50.466352 16526 ts_tablet_manager.cc:1888] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.466499 16526 log.cc:1196] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/fd031f1ac25b42769006740ba81a96b9
I20240502 14:14:50.466933 16526 ts_tablet_manager.cc:1909] T fd031f1ac25b42769006740ba81a96b9 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:50.467025 16527 ts_tablet_manager.cc:1888] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.467828 16527 log.cc:1196] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/257d72583cd94c0d8efdd8925be48dca
I20240502 14:14:50.468482 16527 ts_tablet_manager.cc:1909] T 257d72583cd94c0d8efdd8925be48dca P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:50.469758 16545 log.cc:824] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882: Log is configured to *not* fsync() on all Append() calls
I20240502 14:14:50.470173 16539 tablet_bootstrap.cc:492] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:50.470252 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet fd031f1ac25b42769006740ba81a96b9 (table rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]) successfully deleted
I20240502 14:14:50.470527 16539 ts_tablet_manager.cc:1367] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.003s
I20240502 14:14:50.471175 16527 tablet_replica.cc:330] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:50.471396 16527 raft_consensus.cc:2229] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.471593 16527 raft_consensus.cc:2258] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.471873 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 257d72583cd94c0d8efdd8925be48dca (table rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]) successfully deleted
I20240502 14:14:50.472234 16527 ts_tablet_manager.cc:1875] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.476845 16545 tablet_bootstrap.cc:492] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:50.477113 16545 ts_tablet_manager.cc:1367] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20240502 14:14:50.478530 16539 raft_consensus.cc:348] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:50.478929 16539 raft_consensus.cc:374] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.478986 16527 ts_tablet_manager.cc:1888] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.479063 16539 raft_consensus.cc:729] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.479204 16527 log.cc:1196] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/a5969155f010461293a7f2f61c88c817
I20240502 14:14:50.479656 16527 ts_tablet_manager.cc:1909] T a5969155f010461293a7f2f61c88c817 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:50.479773 16539 consensus_queue.cc:260] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
W20240502 14:14:50.480609 13365 tablet.cc:2369] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:50.484575 16539 ts_tablet_manager.cc:1398] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.014s user 0.008s sys 0.001s
I20240502 14:14:50.484982 16539 tablet_bootstrap.cc:492] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:50.485214 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet a5969155f010461293a7f2f61c88c817 (table rebalance_test_table_0 [id=db2c24362a034dd78c2e5f7797723198]) successfully deleted
I20240502 14:14:50.487099 16539 tablet_bootstrap.cc:654] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.490500 16545 raft_consensus.cc:348] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:50.490912 16545 raft_consensus.cc:374] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.491029 16545 raft_consensus.cc:729] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.491654 16545 consensus_queue.cc:260] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:50.495937 16338 raft_consensus.cc:482] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.496131 16338 raft_consensus.cc:504] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:50.497135 16338 leader_election.cc:288] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:50.510442 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536c22c444344bb485d3e522f0f77154" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:50.510685 13315 raft_consensus.cc:2454] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:50.511181 13384 leader_election.cc:302] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:50.511555 16338 raft_consensus.cc:2781] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:50.511710 16338 raft_consensus.cc:482] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:50.511816 16338 raft_consensus.cc:3037] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.515461 16338 raft_consensus.cc:504] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:50.516335 16338 leader_election.cc:288] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:50.517136 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536c22c444344bb485d3e522f0f77154" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:50.517413 13315 raft_consensus.cc:3037] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.522089 13315 raft_consensus.cc:2454] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:50.522691 13384 leader_election.cc:302] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:50.523078 16338 raft_consensus.cc:2781] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:50.523320 16338 raft_consensus.cc:686] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:50.523556 16338 consensus_queue.cc:237] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:50.537005 13041 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:50.536269 13045 catalog_manager.cc:5484] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:50.552143 16500 tablet.cc:1722] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:50.552907 16506 tablet.cc:1722] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:50.554497 16501 tablet.cc:1722] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240502 14:14:50.559160 13627 tablet.cc:2369] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:50.559042 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536c22c444344bb485d3e522f0f77154" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:14:50.559718 16288 raft_consensus.cc:2454] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:50.585611 13036 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=5d9a14d0d9de4fcdb5f09d781e512363] alter complete (version 3)
I20240502 14:14:50.586923 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "536c22c444344bb485d3e522f0f77154" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:14:50.587139 16288 raft_consensus.cc:3037] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.597164 16288 raft_consensus.cc:2454] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
W20240502 14:14:50.600167 13496 tablet.cc:2369] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:50.603564 16539 tablet_bootstrap.cc:492] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:50.603600 16545 ts_tablet_manager.cc:1398] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.126s user 0.034s sys 0.051s
I20240502 14:14:50.603736 16539 ts_tablet_manager.cc:1367] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.119s user 0.043s sys 0.050s
I20240502 14:14:50.603984 16545 tablet_bootstrap.cc:492] T f64bbd1ae9074071825459e558f03ff5 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:50.604657 16539 raft_consensus.cc:348] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:50.604866 16539 raft_consensus.cc:374] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.604972 16539 raft_consensus.cc:729] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.605180 16539 consensus_queue.cc:260] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:50.605667 16539 ts_tablet_manager.cc:1398] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:50.605732 16545 tablet_bootstrap.cc:654] T f64bbd1ae9074071825459e558f03ff5 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.614714 16545 tablet_bootstrap.cc:492] T f64bbd1ae9074071825459e558f03ff5 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:50.614856 16545 ts_tablet_manager.cc:1367] T f64bbd1ae9074071825459e558f03ff5 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20240502 14:14:50.615478 16545 raft_consensus.cc:348] T f64bbd1ae9074071825459e558f03ff5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:50.615618 16545 raft_consensus.cc:374] T f64bbd1ae9074071825459e558f03ff5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.615718 16545 raft_consensus.cc:729] T f64bbd1ae9074071825459e558f03ff5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.615943 16545 consensus_queue.cc:260] T f64bbd1ae9074071825459e558f03ff5 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:50.616341 16545 ts_tablet_manager.cc:1398] T f64bbd1ae9074071825459e558f03ff5 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20240502 14:14:50.651063 16069 tablet.cc:2369] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:50.701177 13039 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:50.701392 13039 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:50.704269 13039 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 67e2fe12c01e4cf7994eed72f0fa6776
I20240502 14:14:50.705200 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e2fe12c01e4cf7994eed72f0fa6776 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:14:50.705248 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e2fe12c01e4cf7994eed72f0fa6776 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:53312
I20240502 14:14:50.705340 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 67e2fe12c01e4cf7994eed72f0fa6776 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:50.705878 16526 tablet_replica.cc:330] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:50.705960 16592 tablet_replica.cc:330] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:14:50.706108 16526 raft_consensus.cc:2229] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.706305 16526 raft_consensus.cc:2258] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.706375 16592 raft_consensus.cc:2229] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.706972 16592 raft_consensus.cc:2258] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.707198 16526 ts_tablet_manager.cc:1875] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.707782 16591 tablet_replica.cc:330] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:14:50.708187 16591 raft_consensus.cc:2229] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.708514 16591 raft_consensus.cc:2258] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.708561 16592 ts_tablet_manager.cc:1875] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.713624 16526 ts_tablet_manager.cc:1888] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:50.713742 16526 log.cc:1196] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/67e2fe12c01e4cf7994eed72f0fa6776
I20240502 14:14:50.713945 16592 ts_tablet_manager.cc:1888] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:50.714071 16526 ts_tablet_manager.cc:1909] T 67e2fe12c01e4cf7994eed72f0fa6776 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:50.714080 16592 log.cc:1196] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/67e2fe12c01e4cf7994eed72f0fa6776
I20240502 14:14:50.714143 16591 ts_tablet_manager.cc:1875] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.714515 16592 ts_tablet_manager.cc:1909] T 67e2fe12c01e4cf7994eed72f0fa6776 P 8b47d4390f8343a493d6c00123785968: Deleting consensus metadata
I20240502 14:14:50.714905 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 67e2fe12c01e4cf7994eed72f0fa6776 (table rebalancer_dynamic_table_new_name [id=5d9a14d0d9de4fcdb5f09d781e512363]) successfully deleted
I20240502 14:14:50.715896 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 67e2fe12c01e4cf7994eed72f0fa6776 (table rebalancer_dynamic_table_new_name [id=5d9a14d0d9de4fcdb5f09d781e512363]) successfully deleted
I20240502 14:14:50.718739 16591 ts_tablet_manager.cc:1888] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:50.718852 16591 log.cc:1196] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/67e2fe12c01e4cf7994eed72f0fa6776
I20240502 14:14:50.719182 16591 ts_tablet_manager.cc:1909] T 67e2fe12c01e4cf7994eed72f0fa6776 P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:14:50.720461 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 67e2fe12c01e4cf7994eed72f0fa6776 (table rebalancer_dynamic_table_new_name [id=5d9a14d0d9de4fcdb5f09d781e512363]) successfully deleted
I20240502 14:14:50.784786 16490 raft_consensus.cc:482] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.784968 16490 raft_consensus.cc:504] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:50.786587 16490 leader_election.cc:288] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
W20240502 14:14:50.804222 16334 tablet.cc:2369] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:50.809340 13038 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:50.815201 16548 raft_consensus.cc:482] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.815428 16548 raft_consensus.cc:504] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:50.817425 16548 leader_election.cc:288] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:50.824330 16401 tablet_service.cc:1467] Processing CreateTablet for tablet 11c709d69d1d4b4f8f403bec782b5cf5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1d5a8814990647f9af1ee4259dd5a9d8]), partition=
I20240502 14:14:50.824421 16267 tablet_service.cc:1467] Processing CreateTablet for tablet 11c709d69d1d4b4f8f403bec782b5cf5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1d5a8814990647f9af1ee4259dd5a9d8]), partition=
I20240502 14:14:50.824695 16267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c709d69d1d4b4f8f403bec782b5cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.824702 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c709d69d1d4b4f8f403bec782b5cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.830598 16545 tablet_bootstrap.cc:492] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:50.832258 16545 tablet_bootstrap.cc:654] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.833891 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 11c709d69d1d4b4f8f403bec782b5cf5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1d5a8814990647f9af1ee4259dd5a9d8]), partition=
I20240502 14:14:50.834197 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11c709d69d1d4b4f8f403bec782b5cf5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:50.850893 16619 tablet_bootstrap.cc:492] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:50.852785 16619 tablet_bootstrap.cc:654] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.855813 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab73d39b1f7455bba7aef5e515805c7" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:50.856580 16619 tablet_bootstrap.cc:492] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:50.856779 16619 ts_tablet_manager.cc:1367] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240502 14:14:50.857844 16619 raft_consensus.cc:348] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:50.858139 16545 tablet_bootstrap.cc:492] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:50.858343 16545 ts_tablet_manager.cc:1367] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.007s
I20240502 14:14:50.858284 16619 raft_consensus.cc:374] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.858608 16619 raft_consensus.cc:729] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.858990 16619 consensus_queue.cc:260] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:50.859232 16545 raft_consensus.cc:348] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:50.859495 16545 raft_consensus.cc:374] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.859578 16545 raft_consensus.cc:729] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.859807 16545 consensus_queue.cc:260] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:50.860433 16545 ts_tablet_manager.cc:1398] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:50.860848 16617 tablet_bootstrap.cc:492] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:50.863215 16619 ts_tablet_manager.cc:1398] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:50.864673 16617 tablet_bootstrap.cc:654] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:50.868817 15881 raft_consensus.cc:482] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.869099 15881 raft_consensus.cc:504] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:50.870306 15881 leader_election.cc:288] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:50.874994 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eab73d39b1f7455bba7aef5e515805c7" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:50.875315 13579 raft_consensus.cc:2379] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1da7f11f0604f78934354a992bb0f55 in current term 1: Already voted for candidate cd67493f6f0142f896010f2b8a39887d in this term.
I20240502 14:14:50.875911 13038 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 14:14:50.875937 15955 leader_election.cc:302] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d
I20240502 14:14:50.876124 13038 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240502 14:14:50.876363 16548 raft_consensus.cc:3037] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.879120 16548 raft_consensus.cc:2726] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:50.882956 13038 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 2f9a5846fbad47c39a8a197e26b48080
I20240502 14:14:50.883445 13038 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 0395e4ffbe20447d8dbc0722f4c74994
I20240502 14:14:50.883410 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64bbd1ae9074071825459e558f03ff5" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:14:50.883628 16288 raft_consensus.cc:2454] T f64bbd1ae9074071825459e558f03ff5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:14:50.883668 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9a5846fbad47c39a8a197e26b48080 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:50.883909 16529 tablet_replica.cc:330] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:50.884100 16529 raft_consensus.cc:2229] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.884347 16529 raft_consensus.cc:2258] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.885051 16090 leader_election.cc:302] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:50.885360 13429 tablet_service.cc:1514] Processing DeleteTablet for tablet 0395e4ffbe20447d8dbc0722f4c74994 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:50.885612 16529 ts_tablet_manager.cc:1875] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.885771 16490 raft_consensus.cc:2781] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:50.885892 16490 raft_consensus.cc:482] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:50.885987 16527 tablet_replica.cc:330] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:50.885984 16490 raft_consensus.cc:3037] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.886193 16527 raft_consensus.cc:2229] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.886498 16527 raft_consensus.cc:2258] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.887152 16527 ts_tablet_manager.cc:1875] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.888137 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 0395e4ffbe20447d8dbc0722f4c74994 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:50.888728 16490 raft_consensus.cc:504] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:50.889480 16490 leader_election.cc:288] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:50.890033 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64bbd1ae9074071825459e558f03ff5" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:14:50.890215 16288 raft_consensus.cc:3037] T f64bbd1ae9074071825459e558f03ff5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.891369 16529 ts_tablet_manager.cc:1888] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.891487 16529 log.cc:1196] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/2f9a5846fbad47c39a8a197e26b48080
I20240502 14:14:50.891820 16529 ts_tablet_manager.cc:1909] T 2f9a5846fbad47c39a8a197e26b48080 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:50.892499 16288 raft_consensus.cc:2454] T f64bbd1ae9074071825459e558f03ff5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:14:50.892570 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 2f9a5846fbad47c39a8a197e26b48080 (table rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]) successfully deleted
I20240502 14:14:50.892853 16090 leader_election.cc:302] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:50.893034 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9a5846fbad47c39a8a197e26b48080 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:50.893303 16538 tablet_replica.cc:330] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:50.893483 16538 raft_consensus.cc:2229] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.893698 16538 raft_consensus.cc:2258] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.894312 16538 ts_tablet_manager.cc:1875] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.895671 16527 ts_tablet_manager.cc:1888] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.895761 16527 log.cc:1196] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/0395e4ffbe20447d8dbc0722f4c74994
I20240502 14:14:50.896134 13038 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet e293ac3e623c432e8ce57333b74a6386
I20240502 14:14:50.896245 16527 ts_tablet_manager.cc:1909] T 0395e4ffbe20447d8dbc0722f4c74994 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:50.896721 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet e293ac3e623c432e8ce57333b74a6386 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:50.896756 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet e293ac3e623c432e8ce57333b74a6386 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:50.898298 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f9a5846fbad47c39a8a197e26b48080 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:50.898662 16526 tablet_replica.cc:330] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:50.898815 16526 raft_consensus.cc:2229] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.899008 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 0395e4ffbe20447d8dbc0722f4c74994 (table rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]) successfully deleted
I20240502 14:14:50.899168 16526 raft_consensus.cc:2258] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.899938 16526 ts_tablet_manager.cc:1875] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.901173 16538 ts_tablet_manager.cc:1888] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.901453 16490 raft_consensus.cc:2781] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:50.901304 16538 log.cc:1196] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/2f9a5846fbad47c39a8a197e26b48080
I20240502 14:14:50.902439 16538 ts_tablet_manager.cc:1909] T 2f9a5846fbad47c39a8a197e26b48080 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:50.902696 16490 raft_consensus.cc:686] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:14:50.903301 16490 consensus_queue.cc:237] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:50.906162 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 0395e4ffbe20447d8dbc0722f4c74994 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:50.906289 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet e293ac3e623c432e8ce57333b74a6386 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:50 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:50.908633 16526 ts_tablet_manager.cc:1888] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.908978 16526 log.cc:1196] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/2f9a5846fbad47c39a8a197e26b48080
I20240502 14:14:50.909472 16526 ts_tablet_manager.cc:1909] T 2f9a5846fbad47c39a8a197e26b48080 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:50.910161 16529 tablet_replica.cc:330] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:50.910420 16529 raft_consensus.cc:2229] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.910470 16548 raft_consensus.cc:482] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:50.910629 16548 raft_consensus.cc:504] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:50.911456 16548 leader_election.cc:288] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:50.912261 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 2f9a5846fbad47c39a8a197e26b48080 (table rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]) successfully deleted
I20240502 14:14:50.913265 16529 raft_consensus.cc:2258] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.913777 16538 tablet_replica.cc:330] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:50.913894 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 2f9a5846fbad47c39a8a197e26b48080 (table rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]) successfully deleted
I20240502 14:14:50.913651 13038 catalog_manager.cc:5484] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:50.914083 16538 raft_consensus.cc:2229] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:50.914645 16538 raft_consensus.cc:2258] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.915518 16538 ts_tablet_manager.cc:1875] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.917246 16529 ts_tablet_manager.cc:1875] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.917390 16526 tablet_replica.cc:330] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:50.917605 16526 raft_consensus.cc:2229] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.917829 16526 raft_consensus.cc:2258] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.918756 16617 tablet_bootstrap.cc:492] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:50.921469 16617 ts_tablet_manager.cc:1367] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.000s
I20240502 14:14:50.922623 16526 ts_tablet_manager.cc:1875] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.922817 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64bbd1ae9074071825459e558f03ff5" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:14:50.923120 16022 raft_consensus.cc:2454] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:14:50.923491 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64bbd1ae9074071825459e558f03ff5" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:14:50.923717 16023 raft_consensus.cc:3037] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.925868 16617 raft_consensus.cc:348] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:50.926496 16023 raft_consensus.cc:2454] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:14:50.927816 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c709d69d1d4b4f8f403bec782b5cf5" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:14:50.928009 16288 raft_consensus.cc:2454] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:50.928378 16617 raft_consensus.cc:374] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:50.928414 13122 leader_election.cc:302] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 5297e816f782430d9a01220f0471a882; no voters:
I20240502 14:14:50.928550 16617 raft_consensus.cc:729] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:50.928750 15881 raft_consensus.cc:2781] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:50.928812 16617 consensus_queue.cc:260] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:50.928938 15881 raft_consensus.cc:482] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:50.929128 15881 raft_consensus.cc:3037] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.930083 16617 ts_tablet_manager.cc:1398] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:14:50.931246 16531 consensus_queue.cc:1035] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:50.932368 15881 raft_consensus.cc:504] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:50.932643 16526 ts_tablet_manager.cc:1888] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.932782 16526 log.cc:1196] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/0395e4ffbe20447d8dbc0722f4c74994
I20240502 14:14:50.933173 16526 ts_tablet_manager.cc:1909] T 0395e4ffbe20447d8dbc0722f4c74994 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:50.933429 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c709d69d1d4b4f8f403bec782b5cf5" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:14:50.933620 16288 raft_consensus.cc:3037] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.933940 16526 tablet_replica.cc:330] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:50.934151 16526 raft_consensus.cc:2229] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.934168 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 0395e4ffbe20447d8dbc0722f4c74994 (table rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]) successfully deleted
I20240502 14:14:50.934370 16526 raft_consensus.cc:2258] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.934957 16526 ts_tablet_manager.cc:1875] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.936777 16288 raft_consensus.cc:2454] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:50.937376 13122 leader_election.cc:302] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 5297e816f782430d9a01220f0471a882; no voters:
I20240502 14:14:50.937660 16337 raft_consensus.cc:2781] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:50.941241 16337 raft_consensus.cc:686] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:50.941507 16526 ts_tablet_manager.cc:1888] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.941673 16526 log.cc:1196] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/e293ac3e623c432e8ce57333b74a6386
I20240502 14:14:50.941545 16337 consensus_queue.cc:237] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:50.941954 15881 leader_election.cc:288] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:50.942318 16538 ts_tablet_manager.cc:1888] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.942441 16538 log.cc:1196] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/e293ac3e623c432e8ce57333b74a6386
I20240502 14:14:50.942860 16538 ts_tablet_manager.cc:1909] T e293ac3e623c432e8ce57333b74a6386 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:50.943629 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64bbd1ae9074071825459e558f03ff5" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:14:50.943905 16288 raft_consensus.cc:2379] T f64bbd1ae9074071825459e558f03ff5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1da7f11f0604f78934354a992bb0f55 in current term 1: Already voted for candidate c86f7440570f4796babfd44c3d7fd847 in this term.
I20240502 14:14:50.945501 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet e293ac3e623c432e8ce57333b74a6386 (table rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]) successfully deleted
I20240502 14:14:50.949697 16526 ts_tablet_manager.cc:1909] T e293ac3e623c432e8ce57333b74a6386 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:50.950354 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c709d69d1d4b4f8f403bec782b5cf5" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:14:50.950333 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "11c709d69d1d4b4f8f403bec782b5cf5" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:50.950783 16420 raft_consensus.cc:3037] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:50.952378 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet e293ac3e623c432e8ce57333b74a6386 (table rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]) successfully deleted
I20240502 14:14:50.955403 16420 raft_consensus.cc:2454] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:50.957322 13043 catalog_manager.cc:5484] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:50.962294 16529 ts_tablet_manager.cc:1888] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.962460 16529 log.cc:1196] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/0395e4ffbe20447d8dbc0722f4c74994
I20240502 14:14:50.962863 16529 ts_tablet_manager.cc:1909] T 0395e4ffbe20447d8dbc0722f4c74994 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:50.963798 16529 tablet_replica.cc:330] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:50.964017 16529 raft_consensus.cc:2229] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:50.964164 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 0395e4ffbe20447d8dbc0722f4c74994 (table rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]) successfully deleted
I20240502 14:14:50.964315 16529 raft_consensus.cc:2258] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:50.965624 16529 ts_tablet_manager.cc:1875] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:50.969023 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64bbd1ae9074071825459e558f03ff5" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:50.970118 15956 leader_election.cc:302] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:14:50.970718 16548 raft_consensus.cc:2726] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:50.973366 16529 ts_tablet_manager.cc:1888] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:50.973567 16529 log.cc:1196] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/e293ac3e623c432e8ce57333b74a6386
I20240502 14:14:50.974134 16529 ts_tablet_manager.cc:1909] T e293ac3e623c432e8ce57333b74a6386 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:50.975243 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet e293ac3e623c432e8ce57333b74a6386 (table rebalance_test_table_2 [id=c546750f02fa45d7891e9cadb0c3c203]) successfully deleted
I20240502 14:14:50.980201 16528 consensus_queue.cc:1035] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.015918 16267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:51.018849 16401 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:51.029783 13296 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:14:51.030109 16002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:51.037468 13166 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:51.045794 13560 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240502 14:14:51.056351 16135 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240502 14:14:51.056573 16338 consensus_queue.cc:1035] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.067314 16338 consensus_queue.cc:1035] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.075894 13429 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
W20240502 14:14:51.082504 13234 tablet.cc:2369] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:14:51.090250 13043 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:51.172446 16288 raft_consensus.cc:1261] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:51.172701 16420 raft_consensus.cc:1261] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:51.173198 16337 consensus_queue.cc:1035] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.173601 15881 consensus_queue.cc:1035] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.192818 13043 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet 11c709d69d1d4b4f8f403bec782b5cf5 (table rebalancer_dynamic_table [id=1d5a8814990647f9af1ee4259dd5a9d8]). Expected version 1 got 0
I20240502 14:14:51.211692 16660 tablet.cc:1722] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:51.212581 16660 tablet.cc:1620] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:14:51.213935 13043 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet 11c709d69d1d4b4f8f403bec782b5cf5 (table rebalancer_dynamic_table [id=1d5a8814990647f9af1ee4259dd5a9d8]). Expected version 1 got 0
I20240502 14:14:51.214763 16661 tablet.cc:1722] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:51.216279 16661 tablet.cc:1620] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:14:51.217813 13043 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet 11c709d69d1d4b4f8f403bec782b5cf5 (table rebalancer_dynamic_table [id=1d5a8814990647f9af1ee4259dd5a9d8]). Expected version 1 got 0
I20240502 14:14:51.221625 13038 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet 11c709d69d1d4b4f8f403bec782b5cf5 (table rebalancer_dynamic_table [id=1d5a8814990647f9af1ee4259dd5a9d8]). Expected version 1 got 0
I20240502 14:14:51.225142 13043 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=1d5a8814990647f9af1ee4259dd5a9d8] alter complete (version 1)
I20240502 14:14:51.231092 16663 tablet.cc:1722] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:51.231564 16663 tablet.cc:1620] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:14:51.233704 13038 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet 11c709d69d1d4b4f8f403bec782b5cf5 (table rebalancer_dynamic_table [id=1d5a8814990647f9af1ee4259dd5a9d8]). Expected version 1 got 0
I20240502 14:14:51.355624 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:51.366889 16660 tablet.cc:1722] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:51.367380 16660 tablet.cc:1620] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:14:51.367538 16661 tablet.cc:1722] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:51.367931 16661 tablet.cc:1620] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:14:51.368415 16663 tablet.cc:1722] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:51.368881 16663 tablet.cc:1620] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:14:51.375742 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=1d5a8814990647f9af1ee4259dd5a9d8] alter complete (version 2)
I20240502 14:14:51.398664 13038 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 14:14:51.399071 13038 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240502 14:14:51.406921 16490 consensus_queue.cc:1035] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.408406 13038 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 7cc40c95ebb94109a784c659b5b97ffe
I20240502 14:14:51.409052 13038 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet ad1c998f38c049eda8d4ff28fb2eca29
I20240502 14:14:51.409282 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cc40c95ebb94109a784c659b5b97ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:51.409431 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cc40c95ebb94109a784c659b5b97ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:51.409545 16538 tablet_replica.cc:330] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:51.409729 16526 tablet_replica.cc:330] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:51.409746 16538 raft_consensus.cc:2229] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:51.409947 16526 raft_consensus.cc:2229] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.410063 16538 raft_consensus.cc:2258] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.410152 16526 raft_consensus.cc:2258] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.410694 16538 ts_tablet_manager.cc:1875] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.410841 16526 ts_tablet_manager.cc:1875] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.412931 13038 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 9b2f3557f96d4f33b02941d086de1391
I20240502 14:14:51.413195 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1c998f38c049eda8d4ff28fb2eca29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:51.413357 13429 tablet_service.cc:1514] Processing DeleteTablet for tablet 7cc40c95ebb94109a784c659b5b97ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:51.413762 16529 tablet_replica.cc:330] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:51.414041 16529 raft_consensus.cc:2229] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.414269 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2f3557f96d4f33b02941d086de1391 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:51.414352 16529 raft_consensus.cc:2258] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.415115 16529 ts_tablet_manager.cc:1875] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.415205 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2f3557f96d4f33b02941d086de1391 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:51.417510 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1c998f38c049eda8d4ff28fb2eca29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:51.418257 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet ad1c998f38c049eda8d4ff28fb2eca29 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:51.418433 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b2f3557f96d4f33b02941d086de1391 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:51.421578 16698 tablet_replica.cc:330] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:51.421831 16698 raft_consensus.cc:2229] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.422122 16698 raft_consensus.cc:2258] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.422806 16698 ts_tablet_manager.cc:1875] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.424845 16538 ts_tablet_manager.cc:1888] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.425915 16538 log.cc:1196] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/7cc40c95ebb94109a784c659b5b97ffe
I20240502 14:14:51.427289 16538 ts_tablet_manager.cc:1909] T 7cc40c95ebb94109a784c659b5b97ffe P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:51.428544 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 7cc40c95ebb94109a784c659b5b97ffe (table rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]) successfully deleted
I20240502 14:14:51.430995 16538 tablet_replica.cc:330] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:51.431413 16538 raft_consensus.cc:2229] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.431954 16538 raft_consensus.cc:2258] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.432749 16490 consensus_queue.cc:1035] T f64bbd1ae9074071825459e558f03ff5 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.432883 16538 ts_tablet_manager.cc:1875] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.438745 16526 ts_tablet_manager.cc:1888] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.440609 16526 log.cc:1196] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/7cc40c95ebb94109a784c659b5b97ffe
I20240502 14:14:51.442296 16529 ts_tablet_manager.cc:1888] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.442523 16526 ts_tablet_manager.cc:1909] T 7cc40c95ebb94109a784c659b5b97ffe P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:51.443647 16526 tablet_replica.cc:330] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:51.443820 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 7cc40c95ebb94109a784c659b5b97ffe (table rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]) successfully deleted
I20240502 14:14:51.443974 16698 ts_tablet_manager.cc:1888] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.444828 16529 log.cc:1196] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/ad1c998f38c049eda8d4ff28fb2eca29
I20240502 14:14:51.445363 16529 ts_tablet_manager.cc:1909] T ad1c998f38c049eda8d4ff28fb2eca29 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:51.447634 16526 raft_consensus.cc:2229] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:51.450392 16538 ts_tablet_manager.cc:1888] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.450403 16526 raft_consensus.cc:2258] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.451145 16526 ts_tablet_manager.cc:1875] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.451321 16538 log.cc:1196] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/ad1c998f38c049eda8d4ff28fb2eca29
I20240502 14:14:51.452045 16538 ts_tablet_manager.cc:1909] T ad1c998f38c049eda8d4ff28fb2eca29 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:51.452303 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet ad1c998f38c049eda8d4ff28fb2eca29 (table rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]) successfully deleted
I20240502 14:14:51.452373 16529 tablet_replica.cc:330] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:51.452591 16529 raft_consensus.cc:2229] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:51.453117 16529 raft_consensus.cc:2258] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.456115 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet ad1c998f38c049eda8d4ff28fb2eca29 (table rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]) successfully deleted
I20240502 14:14:51.456570 16538 tablet_replica.cc:330] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:51.456969 16538 raft_consensus.cc:2229] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.457319 16538 raft_consensus.cc:2258] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.458114 16538 ts_tablet_manager.cc:1875] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.461896 16529 ts_tablet_manager.cc:1875] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.467408 16538 ts_tablet_manager.cc:1888] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.467574 16538 log.cc:1196] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/9b2f3557f96d4f33b02941d086de1391
I20240502 14:14:51.468156 16538 ts_tablet_manager.cc:1909] T 9b2f3557f96d4f33b02941d086de1391 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:51.469372 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 9b2f3557f96d4f33b02941d086de1391 (table rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]) successfully deleted
I20240502 14:14:51.472133 16698 log.cc:1196] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/7cc40c95ebb94109a784c659b5b97ffe
I20240502 14:14:51.472512 16698 ts_tablet_manager.cc:1909] T 7cc40c95ebb94109a784c659b5b97ffe P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:51.473268 16698 tablet_replica.cc:330] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:51.473418 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 7cc40c95ebb94109a784c659b5b97ffe (table rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]) successfully deleted
I20240502 14:14:51.477252 16698 raft_consensus.cc:2229] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.477540 16698 raft_consensus.cc:2258] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.478297 16698 ts_tablet_manager.cc:1875] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.485764 16698 ts_tablet_manager.cc:1888] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.485778 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:51.485965 16698 log.cc:1196] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/9b2f3557f96d4f33b02941d086de1391
I20240502 14:14:51.486450 16698 ts_tablet_manager.cc:1909] T 9b2f3557f96d4f33b02941d086de1391 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:51.492311 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 9b2f3557f96d4f33b02941d086de1391 (table rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]) successfully deleted
I20240502 14:14:51.496438 16660 tablet.cc:1722] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:51.497829 16663 tablet.cc:1722] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:51.503006 16661 tablet.cc:1722] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:51.505808 13040 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=1d5a8814990647f9af1ee4259dd5a9d8] alter complete (version 3)
I20240502 14:14:51.509568 16529 ts_tablet_manager.cc:1888] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.509761 16529 log.cc:1196] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/9b2f3557f96d4f33b02941d086de1391
I20240502 14:14:51.510183 16529 ts_tablet_manager.cc:1909] T 9b2f3557f96d4f33b02941d086de1391 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:51.510428 16526 ts_tablet_manager.cc:1888] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.511804 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 9b2f3557f96d4f33b02941d086de1391 (table rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]) successfully deleted
I20240502 14:14:51.511986 16526 log.cc:1196] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/ad1c998f38c049eda8d4ff28fb2eca29
I20240502 14:14:51.512411 16526 ts_tablet_manager.cc:1909] T ad1c998f38c049eda8d4ff28fb2eca29 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:51.514628 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet ad1c998f38c049eda8d4ff28fb2eca29 (table rebalance_test_table_4 [id=bcd8d28081a34aee9b452ec54a7b956c]) successfully deleted
I20240502 14:14:51.515278 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:51.525144 13560 tablet_service.cc:1467] Processing CreateTablet for tablet f961cf70d8ca482ba331241052833fd3 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7a3f9a47775b46e38274ca5c7b2dbe6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.525285 16267 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab40fe08f714c99a5961cc2ad0915d3 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7a3f9a47775b46e38274ca5c7b2dbe6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.525580 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f961cf70d8ca482ba331241052833fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.525686 16267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab40fe08f714c99a5961cc2ad0915d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.528532 16401 tablet_service.cc:1467] Processing CreateTablet for tablet f961cf70d8ca482ba331241052833fd3 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7a3f9a47775b46e38274ca5c7b2dbe6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.529093 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f961cf70d8ca482ba331241052833fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.530079 16135 tablet_service.cc:1467] Processing CreateTablet for tablet f961cf70d8ca482ba331241052833fd3 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7a3f9a47775b46e38274ca5c7b2dbe6d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.530500 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f961cf70d8ca482ba331241052833fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.535624 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab40fe08f714c99a5961cc2ad0915d3 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7a3f9a47775b46e38274ca5c7b2dbe6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.536051 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab40fe08f714c99a5961cc2ad0915d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.536177 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2b8c72c7ba4be58f31ac073e41f2a8 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7a3f9a47775b46e38274ca5c7b2dbe6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.536463 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2b8c72c7ba4be58f31ac073e41f2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.539456 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2b8c72c7ba4be58f31ac073e41f2a8 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7a3f9a47775b46e38274ca5c7b2dbe6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.539880 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2b8c72c7ba4be58f31ac073e41f2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.540026 16400 tablet_service.cc:1467] Processing CreateTablet for tablet 3ab40fe08f714c99a5961cc2ad0915d3 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7a3f9a47775b46e38274ca5c7b2dbe6d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.540445 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab40fe08f714c99a5961cc2ad0915d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.544275 16710 tablet_bootstrap.cc:492] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:51.545547 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 1e2b8c72c7ba4be58f31ac073e41f2a8 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=7a3f9a47775b46e38274ca5c7b2dbe6d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:51.545863 16710 tablet_bootstrap.cc:654] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.545918 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2b8c72c7ba4be58f31ac073e41f2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.557552 16713 tablet_bootstrap.cc:492] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:51.557924 16715 tablet_bootstrap.cc:492] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:51.559459 16715 tablet_bootstrap.cc:654] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.559579 16713 tablet_bootstrap.cc:654] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.560899 16712 tablet_bootstrap.cc:492] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:51.561453 16711 tablet_bootstrap.cc:492] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:51.565704 16712 tablet_bootstrap.cc:654] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.566228 16710 tablet_bootstrap.cc:492] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:51.566327 16711 tablet_bootstrap.cc:654] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.566421 16710 ts_tablet_manager.cc:1367] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20240502 14:14:51.567400 16710 raft_consensus.cc:348] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.567762 16710 raft_consensus.cc:374] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.567934 16710 raft_consensus.cc:729] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.567991 16715 tablet_bootstrap.cc:492] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:51.568154 16715 ts_tablet_manager.cc:1367] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240502 14:14:51.569078 16715 raft_consensus.cc:348] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:51.569428 16715 raft_consensus.cc:374] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.569612 16715 raft_consensus.cc:729] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.569880 16715 consensus_queue.cc:260] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:51.570662 16715 ts_tablet_manager.cc:1398] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:51.570706 16711 tablet_bootstrap.cc:492] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:51.570822 16711 ts_tablet_manager.cc:1367] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240502 14:14:51.571549 16711 raft_consensus.cc:348] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:51.571712 16711 raft_consensus.cc:374] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.571849 16710 consensus_queue.cc:260] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.572333 16713 tablet_bootstrap.cc:492] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:51.572458 16713 ts_tablet_manager.cc:1367] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20240502 14:14:51.573154 16713 raft_consensus.cc:348] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.573397 16713 raft_consensus.cc:374] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.573496 16710 ts_tablet_manager.cc:1398] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 14:14:51.573519 16713 raft_consensus.cc:729] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.573758 16713 consensus_queue.cc:260] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.574338 16713 ts_tablet_manager.cc:1398] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:51.575310 16713 tablet_bootstrap.cc:492] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:51.577143 16713 tablet_bootstrap.cc:654] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.578076 16711 raft_consensus.cc:729] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.579453 16712 tablet_bootstrap.cc:492] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:51.579680 16712 ts_tablet_manager.cc:1367] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240502 14:14:51.581159 16712 raft_consensus.cc:348] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.581485 16712 raft_consensus.cc:374] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.581420 16711 consensus_queue.cc:260] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:51.581670 16712 raft_consensus.cc:729] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.582008 16711 ts_tablet_manager.cc:1398] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 14:14:51.581924 16712 consensus_queue.cc:260] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.582713 16712 ts_tablet_manager.cc:1398] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:51.583184 16712 tablet_bootstrap.cc:492] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:51.584700 16712 tablet_bootstrap.cc:654] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.584931 16713 tablet_bootstrap.cc:492] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:51.585112 16713 ts_tablet_manager.cc:1367] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240502 14:14:51.586138 16711 tablet_bootstrap.cc:492] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:51.588130 16711 tablet_bootstrap.cc:654] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.591187 16713 raft_consensus.cc:348] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:51.592082 16713 raft_consensus.cc:374] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.592243 16713 raft_consensus.cc:729] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.592618 16713 consensus_queue.cc:260] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:51.593925 16713 ts_tablet_manager.cc:1398] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:14:51.594607 16712 tablet_bootstrap.cc:492] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:51.594827 16712 ts_tablet_manager.cc:1367] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20240502 14:14:51.595872 16712 raft_consensus.cc:348] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.596140 16712 raft_consensus.cc:374] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.596273 16712 raft_consensus.cc:729] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.596583 16712 consensus_queue.cc:260] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.597128 16712 ts_tablet_manager.cc:1398] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:51.609395 16711 tablet_bootstrap.cc:492] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:51.609433 13046 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:51.609725 16711 ts_tablet_manager.cc:1367] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.007s
I20240502 14:14:51.609748 13046 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:51.610697 16711 raft_consensus.cc:348] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.611020 16711 raft_consensus.cc:374] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.611131 16711 raft_consensus.cc:729] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.611483 16711 consensus_queue.cc:260] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.613174 16711 ts_tablet_manager.cc:1398] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:51.613232 16704 raft_consensus.cc:482] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:51.613548 16711 tablet_bootstrap.cc:492] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:51.613374 16704 raft_consensus.cc:504] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.613692 13046 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 11c709d69d1d4b4f8f403bec782b5cf5
I20240502 14:14:51.614382 16704 leader_election.cc:288] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:51.614580 16267 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c709d69d1d4b4f8f403bec782b5cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:14:51.614645 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c709d69d1d4b4f8f403bec782b5cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:51.615101 16526 tablet_replica.cc:330] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:51.615126 16493 raft_consensus.cc:482] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:51.615305 16526 raft_consensus.cc:2229] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:51.615289 16493 raft_consensus.cc:504] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:51.615643 16526 raft_consensus.cc:2258] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.615922 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet 11c709d69d1d4b4f8f403bec782b5cf5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:53312
I20240502 14:14:51.616060 16493 leader_election.cc:288] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:51.617586 16526 ts_tablet_manager.cc:1875] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.617978 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f961cf70d8ca482ba331241052833fd3" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:51.618187 16155 raft_consensus.cc:2454] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:14:51.620326 16354 leader_election.cc:302] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:51.621950 16493 raft_consensus.cc:2781] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:51.622051 16493 raft_consensus.cc:482] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:51.622124 16493 raft_consensus.cc:3037] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.625327 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2b8c72c7ba4be58f31ac073e41f2a8" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:51.625569 13187 raft_consensus.cc:2454] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:14:51.626147 16088 leader_election.cc:302] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:51.626384 16704 raft_consensus.cc:2781] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:51.626519 16704 raft_consensus.cc:482] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:51.626626 16704 raft_consensus.cc:3037] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.628522 16704 raft_consensus.cc:504] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.629050 16704 leader_election.cc:288] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:51.629501 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2b8c72c7ba4be58f31ac073e41f2a8" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:51.629738 13187 raft_consensus.cc:3037] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.631997 16711 tablet_bootstrap.cc:654] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.631876 16493 raft_consensus.cc:504] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:51.631987 16526 ts_tablet_manager.cc:1888] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:51.632122 16526 log.cc:1196] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/11c709d69d1d4b4f8f403bec782b5cf5
I20240502 14:14:51.632179 13187 raft_consensus.cc:2454] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:14:51.632514 16526 ts_tablet_manager.cc:1909] T 11c709d69d1d4b4f8f403bec782b5cf5 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:51.632957 16493 leader_election.cc:288] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:51.633600 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f961cf70d8ca482ba331241052833fd3" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:51.633823 16155 raft_consensus.cc:3037] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.636448 16155 raft_consensus.cc:2454] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:14:51.639205 16088 leader_election.cc:302] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:51.639755 16354 leader_election.cc:302] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:51.640472 16704 raft_consensus.cc:2781] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:51.641708 16704 raft_consensus.cc:686] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:14:51.642058 16704 consensus_queue.cc:237] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.642458 16493 raft_consensus.cc:2781] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:51.651063 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 11c709d69d1d4b4f8f403bec782b5cf5 (table rebalancer_dynamic_table_new_name [id=1d5a8814990647f9af1ee4259dd5a9d8]) successfully deleted
I20240502 14:14:51.653539 13046 catalog_manager.cc:5484] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:51.655648 16711 tablet_bootstrap.cc:492] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:51.657019 16711 ts_tablet_manager.cc:1367] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.001s
I20240502 14:14:51.658106 16735 tablet_replica.cc:330] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:14:51.658043 16711 raft_consensus.cc:348] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.658268 16711 raft_consensus.cc:374] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.658401 16711 raft_consensus.cc:729] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.658514 16735 raft_consensus.cc:2229] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.661293 16735 raft_consensus.cc:2258] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.660768 16711 consensus_queue.cc:260] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.663197 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f961cf70d8ca482ba331241052833fd3" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:51.663435 13579 raft_consensus.cc:2454] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:14:51.663516 16732 tablet_replica.cc:330] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:14:51.663494 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f961cf70d8ca482ba331241052833fd3" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:51.663724 13581 raft_consensus.cc:3037] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.663985 16732 raft_consensus.cc:2229] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.664292 16732 raft_consensus.cc:2258] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.664820 16711 ts_tablet_manager.cc:1398] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 14:14:51.665079 16735 ts_tablet_manager.cc:1875] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.666795 16732 ts_tablet_manager.cc:1875] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.670892 16735 ts_tablet_manager.cc:1888] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:51.671037 16735 log.cc:1196] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/11c709d69d1d4b4f8f403bec782b5cf5
I20240502 14:14:51.671452 16735 ts_tablet_manager.cc:1909] T 11c709d69d1d4b4f8f403bec782b5cf5 P 8b47d4390f8343a493d6c00123785968: Deleting consensus metadata
I20240502 14:14:51.674304 13581 raft_consensus.cc:2454] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:14:51.675716 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 11c709d69d1d4b4f8f403bec782b5cf5 (table rebalancer_dynamic_table_new_name [id=1d5a8814990647f9af1ee4259dd5a9d8]) successfully deleted
I20240502 14:14:51.677894 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2b8c72c7ba4be58f31ac073e41f2a8" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:51.678010 16732 ts_tablet_manager.cc:1888] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:51.678087 16420 raft_consensus.cc:2454] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:14:51.678305 16732 log.cc:1196] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/11c709d69d1d4b4f8f403bec782b5cf5
I20240502 14:14:51.678364 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2b8c72c7ba4be58f31ac073e41f2a8" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:14:51.678577 16421 raft_consensus.cc:3037] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.679132 16732 ts_tablet_manager.cc:1909] T 11c709d69d1d4b4f8f403bec782b5cf5 P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:14:51.681133 16421 raft_consensus.cc:2454] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:14:51.683852 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 11c709d69d1d4b4f8f403bec782b5cf5 (table rebalancer_dynamic_table_new_name [id=1d5a8814990647f9af1ee4259dd5a9d8]) successfully deleted
I20240502 14:14:51.685927 16738 raft_consensus.cc:482] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:51.686097 16738 raft_consensus.cc:504] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.687201 16738 leader_election.cc:288] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:51.687677 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab40fe08f714c99a5961cc2ad0915d3" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:51.687898 13187 raft_consensus.cc:2454] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:14:51.689291 16353 leader_election.cc:302] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:14:51.691937 16267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:51.695067 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab40fe08f714c99a5961cc2ad0915d3" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:14:51.695328 16288 raft_consensus.cc:2454] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:14:51.696717 16738 raft_consensus.cc:2781] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:51.696833 16738 raft_consensus.cc:482] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:51.696903 16738 raft_consensus.cc:3037] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.700296 16738 raft_consensus.cc:504] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.701193 16493 raft_consensus.cc:686] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:14:51.701738 16493 consensus_queue.cc:237] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:51.703521 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab40fe08f714c99a5961cc2ad0915d3" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:14:51.703754 16288 raft_consensus.cc:3037] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.706547 13046 catalog_manager.cc:5484] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:51.706929 16288 raft_consensus.cc:2454] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:14:51.711731 16355 leader_election.cc:302] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:14:51.712121 16493 raft_consensus.cc:2781] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:51.712278 16493 raft_consensus.cc:686] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:14:51.712554 16493 consensus_queue.cc:237] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:51.716773 13168 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:14:51.726011 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:51.727145 13044 catalog_manager.cc:5484] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:51.729175 16738 leader_election.cc:288] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:51.730454 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ab40fe08f714c99a5961cc2ad0915d3" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:51.730691 13187 raft_consensus.cc:3037] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.736510 13560 tablet_service.cc:1467] Processing CreateTablet for tablet b78249b66e93406cb271fcbbc217ecea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=53e326f05b5545f58887e5f17abc4ab7]), partition=
I20240502 14:14:51.736860 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78249b66e93406cb271fcbbc217ecea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.738572 16267 tablet_service.cc:1467] Processing CreateTablet for tablet b78249b66e93406cb271fcbbc217ecea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=53e326f05b5545f58887e5f17abc4ab7]), partition=
I20240502 14:14:51.738947 16267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78249b66e93406cb271fcbbc217ecea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.744480 13187 raft_consensus.cc:2454] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:14:51.744791 16715 tablet_bootstrap.cc:492] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:51.746161 16715 tablet_bootstrap.cc:654] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.745365 13168 tablet_service.cc:1467] Processing CreateTablet for tablet b78249b66e93406cb271fcbbc217ecea (DEFAULT_TABLE table=rebalancer_dynamic_table [id=53e326f05b5545f58887e5f17abc4ab7]), partition=
I20240502 14:14:51.748957 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b78249b66e93406cb271fcbbc217ecea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:51.751153 16401 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:51.751821 13559 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:14:51.755992 16712 tablet_bootstrap.cc:492] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:51.757539 16712 tablet_bootstrap.cc:654] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.761392 16134 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:51.764441 16710 tablet_bootstrap.cc:492] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:51.765826 16710 tablet_bootstrap.cc:654] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:51.772848 16715 tablet_bootstrap.cc:492] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:51.773056 16715 ts_tablet_manager.cc:1367] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.003s
I20240502 14:14:51.773880 16715 raft_consensus.cc:348] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:51.774190 16715 raft_consensus.cc:374] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.774328 16715 raft_consensus.cc:729] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.774622 16715 consensus_queue.cc:260] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:51.775290 16715 ts_tablet_manager.cc:1398] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:51.777721 16712 tablet_bootstrap.cc:492] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:51.777909 16712 ts_tablet_manager.cc:1367] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20240502 14:14:51.778800 16712 raft_consensus.cc:348] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:51.779067 16712 raft_consensus.cc:374] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.781499 16712 raft_consensus.cc:729] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.781787 16712 consensus_queue.cc:260] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:51.783417 16712 ts_tablet_manager.cc:1398] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:14:51.787039 16710 tablet_bootstrap.cc:492] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:51.787249 16710 ts_tablet_manager.cc:1367] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20240502 14:14:51.787955 16710 raft_consensus.cc:348] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:51.788712 16710 raft_consensus.cc:374] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:51.788844 16710 raft_consensus.cc:729] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:51.789173 16710 consensus_queue.cc:260] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:51.789990 16710 ts_tablet_manager.cc:1398] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:51.790871 13429 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:14:51.833056 13298 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:14:51.845934 16337 raft_consensus.cc:482] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:51.846105 16337 raft_consensus.cc:504] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:51.846899 16337 leader_election.cc:288] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:14:51.847270 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78249b66e93406cb271fcbbc217ecea" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:51.847497 13581 raft_consensus.cc:2454] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:51.847497 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78249b66e93406cb271fcbbc217ecea" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:14:51.847702 16288 raft_consensus.cc:2454] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:51.848037 13120 leader_election.cc:302] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:51.850442 16337 raft_consensus.cc:2781] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:51.850576 16337 raft_consensus.cc:482] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:51.850657 16337 raft_consensus.cc:3037] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.852486 16337 raft_consensus.cc:504] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:51.853334 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78249b66e93406cb271fcbbc217ecea" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:51.853549 13581 raft_consensus.cc:3037] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.855067 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b78249b66e93406cb271fcbbc217ecea" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:14:51.855284 16288 raft_consensus.cc:3037] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:51.856662 13581 raft_consensus.cc:2454] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:51.857075 13120 leader_election.cc:302] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:51.857425 15881 raft_consensus.cc:2781] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:51.857815 15881 raft_consensus.cc:686] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:51.858095 15881 consensus_queue.cc:237] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:51.861562 13046 catalog_manager.cc:5484] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:51.864248 16288 raft_consensus.cc:2454] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:51.865530 16337 leader_election.cc:288] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:14:51.904086 16002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:51.920202 13046 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 14:14:51.920405 13046 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240502 14:14:51.924168 13046 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet c1edaea0b866430ba8c5bd6bac5a9b66
I20240502 14:14:51.924590 13046 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet c54b3251d0ff4e48b93b95634b589b4c
I20240502 14:14:51.924901 13046 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 8e96c49666584fceb759383d24ac1c4a
I20240502 14:14:51.925177 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet c54b3251d0ff4e48b93b95634b589b4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:51.925293 13429 tablet_service.cc:1514] Processing DeleteTablet for tablet c54b3251d0ff4e48b93b95634b589b4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:51.925428 16529 tablet_replica.cc:330] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:51.925525 16698 tablet_replica.cc:330] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:51.925642 16529 raft_consensus.cc:2229] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.925709 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet c1edaea0b866430ba8c5bd6bac5a9b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:51.925709 16698 raft_consensus.cc:2229] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.925871 16529 raft_consensus.cc:2258] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.925902 16698 raft_consensus.cc:2258] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.925997 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet c1edaea0b866430ba8c5bd6bac5a9b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:51.926252 16526 tablet_replica.cc:330] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:51.926432 16526 raft_consensus.cc:2229] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.926473 16698 ts_tablet_manager.cc:1875] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.926543 16529 ts_tablet_manager.cc:1875] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.926651 16526 raft_consensus.cc:2258] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.926719 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet c1edaea0b866430ba8c5bd6bac5a9b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:51.926985 16538 tablet_replica.cc:330] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:51.927201 16538 raft_consensus.cc:2229] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.927258 16526 ts_tablet_manager.cc:1875] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.927407 16538 raft_consensus.cc:2258] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.928061 16538 ts_tablet_manager.cc:1875] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.930280 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet c54b3251d0ff4e48b93b95634b589b4c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:51.930692 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e96c49666584fceb759383d24ac1c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:51.932305 16529 ts_tablet_manager.cc:1888] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.932410 16529 log.cc:1196] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/c54b3251d0ff4e48b93b95634b589b4c
I20240502 14:14:51.932742 16529 ts_tablet_manager.cc:1909] T c54b3251d0ff4e48b93b95634b589b4c P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:51.933929 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet c54b3251d0ff4e48b93b95634b589b4c (table rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]) successfully deleted
I20240502 14:14:51.933966 16526 ts_tablet_manager.cc:1888] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.934064 16526 log.cc:1196] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/c1edaea0b866430ba8c5bd6bac5a9b66
I20240502 14:14:51.934068 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e96c49666584fceb759383d24ac1c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:51.934191 16529 tablet_replica.cc:330] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:51.934357 16529 raft_consensus.cc:2229] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:51.934453 16526 ts_tablet_manager.cc:1909] T c1edaea0b866430ba8c5bd6bac5a9b66 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:51.934513 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e96c49666584fceb759383d24ac1c4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:51 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:51.934759 16529 raft_consensus.cc:2258] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.935200 16526 tablet_replica.cc:330] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:51.935245 16698 ts_tablet_manager.cc:1888] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.935328 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet c1edaea0b866430ba8c5bd6bac5a9b66 (table rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]) successfully deleted
I20240502 14:14:51.935407 16526 raft_consensus.cc:2229] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.935425 16529 ts_tablet_manager.cc:1875] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.935451 16698 log.cc:1196] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/c54b3251d0ff4e48b93b95634b589b4c
I20240502 14:14:51.935608 16526 raft_consensus.cc:2258] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.935634 16538 ts_tablet_manager.cc:1888] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.935761 16538 log.cc:1196] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/c1edaea0b866430ba8c5bd6bac5a9b66
I20240502 14:14:51.936094 16698 ts_tablet_manager.cc:1909] T c54b3251d0ff4e48b93b95634b589b4c P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:51.936113 16538 ts_tablet_manager.cc:1909] T c1edaea0b866430ba8c5bd6bac5a9b66 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:51.936281 16526 ts_tablet_manager.cc:1875] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.936929 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet c1edaea0b866430ba8c5bd6bac5a9b66 (table rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]) successfully deleted
I20240502 14:14:51.937132 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet c54b3251d0ff4e48b93b95634b589b4c (table rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]) successfully deleted
I20240502 14:14:51.937243 16538 tablet_replica.cc:330] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:51.937484 16538 raft_consensus.cc:2229] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:51.937801 16538 raft_consensus.cc:2258] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.938494 16538 ts_tablet_manager.cc:1875] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.940737 16529 ts_tablet_manager.cc:1888] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.940917 16529 log.cc:1196] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/c1edaea0b866430ba8c5bd6bac5a9b66
I20240502 14:14:51.941354 16529 ts_tablet_manager.cc:1909] T c1edaea0b866430ba8c5bd6bac5a9b66 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:51.941900 16526 ts_tablet_manager.cc:1888] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.942054 16526 log.cc:1196] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/8e96c49666584fceb759383d24ac1c4a
I20240502 14:14:51.942374 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet c1edaea0b866430ba8c5bd6bac5a9b66 (table rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]) successfully deleted
I20240502 14:14:51.942407 16529 tablet_replica.cc:330] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:51.942464 16526 ts_tablet_manager.cc:1909] T 8e96c49666584fceb759383d24ac1c4a P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:51.942685 16529 raft_consensus.cc:2229] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:51.942999 16529 raft_consensus.cc:2258] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.943256 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 8e96c49666584fceb759383d24ac1c4a (table rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]) successfully deleted
I20240502 14:14:51.943681 16529 ts_tablet_manager.cc:1875] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.949338 16529 ts_tablet_manager.cc:1888] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.949472 16529 log.cc:1196] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/8e96c49666584fceb759383d24ac1c4a
I20240502 14:14:51.949857 16529 ts_tablet_manager.cc:1909] T 8e96c49666584fceb759383d24ac1c4a P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:51.950644 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 8e96c49666584fceb759383d24ac1c4a (table rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]) successfully deleted
I20240502 14:14:51.951387 16538 ts_tablet_manager.cc:1888] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.951514 16538 log.cc:1196] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/c54b3251d0ff4e48b93b95634b589b4c
I20240502 14:14:51.951860 16538 ts_tablet_manager.cc:1909] T c54b3251d0ff4e48b93b95634b589b4c P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:51.952483 16538 tablet_replica.cc:330] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:51.952566 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet c54b3251d0ff4e48b93b95634b589b4c (table rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]) successfully deleted
I20240502 14:14:51.952711 16538 raft_consensus.cc:2229] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:51.952896 16538 raft_consensus.cc:2258] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:51.953497 16538 ts_tablet_manager.cc:1875] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:51.957872 16538 ts_tablet_manager.cc:1888] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:51.957993 16538 log.cc:1196] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/8e96c49666584fceb759383d24ac1c4a
I20240502 14:14:51.958334 16538 ts_tablet_manager.cc:1909] T 8e96c49666584fceb759383d24ac1c4a P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:51.959081 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 8e96c49666584fceb759383d24ac1c4a (table rebalance_test_table_6 [id=6607ab63fd0e452e8f7c02918e77ea9e]) successfully deleted
I20240502 14:14:51.967089 13046 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:51.972652 16288 raft_consensus.cc:1261] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:51.972884 13581 raft_consensus.cc:1261] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:51.973286 16337 consensus_queue.cc:1035] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.973593 15881 consensus_queue.cc:1035] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:51.976879 13046 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet b78249b66e93406cb271fcbbc217ecea (table rebalancer_dynamic_table [id=53e326f05b5545f58887e5f17abc4ab7]). Expected version 1 got 0
I20240502 14:14:51.978860 16660 tablet.cc:1722] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:51.979348 16660 tablet.cc:1620] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:14:51.984215 16661 tablet.cc:1722] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:51.984699 16661 tablet.cc:1620] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:14:51.985052 16655 mvcc.cc:205] Tried to move back new op lower bound from 7023244459913342976 to 7023244459452997632. Current Snapshot: MvccSnapshot[applied={T|T < 7023244459913342976}]
I20240502 14:14:51.985419 13044 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet b78249b66e93406cb271fcbbc217ecea (table rebalancer_dynamic_table [id=53e326f05b5545f58887e5f17abc4ab7]). Expected version 1 got 0
I20240502 14:14:51.988399 13046 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=53e326f05b5545f58887e5f17abc4ab7] alter complete (version 1)
I20240502 14:14:51.997736 13044 catalog_manager.cc:5576] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817) does not have the latest schema for tablet b78249b66e93406cb271fcbbc217ecea (table rebalancer_dynamic_table [id=53e326f05b5545f58887e5f17abc4ab7]). Expected version 1 got 0
I20240502 14:14:52.000686 16760 tablet.cc:1722] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:52.002012 16760 tablet.cc:1620] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: MemRowSet was empty: no flush needed.
I20240502 14:14:52.002431 13044 catalog_manager.cc:5576] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817) does not have the latest schema for tablet b78249b66e93406cb271fcbbc217ecea (table rebalancer_dynamic_table [id=53e326f05b5545f58887e5f17abc4ab7]). Expected version 1 got 0
I20240502 14:14:52.094247 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:52.101375 16660 tablet.cc:1722] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:52.101855 16660 tablet.cc:1620] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:14:52.102609 16661 tablet.cc:1722] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:52.103041 16661 tablet.cc:1620] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:14:52.107909 16760 tablet.cc:1722] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:52.108449 16760 tablet.cc:1620] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: MemRowSet was empty: no flush needed.
I20240502 14:14:52.115202 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=53e326f05b5545f58887e5f17abc4ab7] alter complete (version 2)
I20240502 14:14:52.115852 16738 consensus_queue.cc:1035] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.126173 16738 consensus_queue.cc:1035] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.191496 16738 consensus_queue.cc:1035] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.200582 16741 consensus_queue.cc:1035] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.205905 16704 consensus_queue.cc:1035] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.212985 16704 consensus_queue.cc:1035] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.225204 13046 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:52.235677 16660 tablet.cc:1722] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:52.236299 16760 tablet.cc:1722] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:52.242637 16661 tablet.cc:1722] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:52.246687 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:52.262387 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 16d6faae2add42c5bd4af6018a13b841 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cb136ab1e407401daef8e9856e8ee6f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.262817 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 16d6faae2add42c5bd4af6018a13b841 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cb136ab1e407401daef8e9856e8ee6f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.262933 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d6faae2add42c5bd4af6018a13b841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.263298 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d6faae2add42c5bd4af6018a13b841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.266625 16401 tablet_service.cc:1467] Processing CreateTablet for tablet 7da975869e82479b91fdda30680e6b3c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cb136ab1e407401daef8e9856e8ee6f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.267117 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da975869e82479b91fdda30680e6b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.271335 16715 tablet_bootstrap.cc:492] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:52.272949 16715 tablet_bootstrap.cc:654] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.274542 16267 tablet_service.cc:1467] Processing CreateTablet for tablet 7da975869e82479b91fdda30680e6b3c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cb136ab1e407401daef8e9856e8ee6f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.274816 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 16d6faae2add42c5bd4af6018a13b841 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cb136ab1e407401daef8e9856e8ee6f9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.274984 16267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da975869e82479b91fdda30680e6b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.275229 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d6faae2add42c5bd4af6018a13b841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.275313 13040 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=53e326f05b5545f58887e5f17abc4ab7] alter complete (version 3)
I20240502 14:14:52.275489 16265 tablet_service.cc:1467] Processing CreateTablet for tablet caa4af5f41674aaa8587158f25e495fb (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cb136ab1e407401daef8e9856e8ee6f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.275866 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa4af5f41674aaa8587158f25e495fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.277110 16002 tablet_service.cc:1467] Processing CreateTablet for tablet 7da975869e82479b91fdda30680e6b3c (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cb136ab1e407401daef8e9856e8ee6f9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.277523 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da975869e82479b91fdda30680e6b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.282343 16710 tablet_bootstrap.cc:492] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:52.283851 16710 tablet_bootstrap.cc:654] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.290593 13427 tablet_service.cc:1467] Processing CreateTablet for tablet caa4af5f41674aaa8587158f25e495fb (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cb136ab1e407401daef8e9856e8ee6f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.290746 16003 tablet_service.cc:1467] Processing CreateTablet for tablet caa4af5f41674aaa8587158f25e495fb (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=cb136ab1e407401daef8e9856e8ee6f9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:52.290951 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa4af5f41674aaa8587158f25e495fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.291110 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa4af5f41674aaa8587158f25e495fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.309355 16710 tablet_bootstrap.cc:492] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:52.309571 16710 ts_tablet_manager.cc:1367] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.007s
I20240502 14:14:52.310694 16710 raft_consensus.cc:348] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.311050 16710 raft_consensus.cc:374] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.311232 16710 raft_consensus.cc:729] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.311573 16710 consensus_queue.cc:260] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.312312 16710 ts_tablet_manager.cc:1398] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:52.312841 16710 tablet_bootstrap.cc:492] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:52.315315 16710 tablet_bootstrap.cc:654] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.326952 16808 tablet_bootstrap.cc:492] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:52.327458 16809 tablet_bootstrap.cc:492] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:52.328975 16710 tablet_bootstrap.cc:492] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:52.329154 16710 ts_tablet_manager.cc:1367] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20240502 14:14:52.329890 16710 raft_consensus.cc:348] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.330096 16710 raft_consensus.cc:374] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.330176 16710 raft_consensus.cc:729] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.330441 16710 consensus_queue.cc:260] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.330758 16715 tablet_bootstrap.cc:492] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:52.330987 16715 ts_tablet_manager.cc:1367] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.003s
I20240502 14:14:52.331810 16715 raft_consensus.cc:348] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.332090 16715 raft_consensus.cc:374] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.332201 16715 raft_consensus.cc:729] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.332453 16715 consensus_queue.cc:260] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.333269 16715 ts_tablet_manager.cc:1398] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:52.333591 16710 ts_tablet_manager.cc:1398] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:52.334048 16809 tablet_bootstrap.cc:654] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.335258 16807 tablet_bootstrap.cc:492] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:52.332643 16808 tablet_bootstrap.cc:654] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.336884 16710 tablet_bootstrap.cc:492] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:52.337649 16807 tablet_bootstrap.cc:654] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.338461 16710 tablet_bootstrap.cc:654] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.342707 16808 tablet_bootstrap.cc:492] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:52.342867 16808 ts_tablet_manager.cc:1367] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240502 14:14:52.343521 16808 raft_consensus.cc:348] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.343755 16808 raft_consensus.cc:374] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.343866 16808 raft_consensus.cc:729] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.344091 16808 consensus_queue.cc:260] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.344744 16808 ts_tablet_manager.cc:1398] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:52.345099 16808 tablet_bootstrap.cc:492] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:52.347877 16808 tablet_bootstrap.cc:654] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.353462 16807 tablet_bootstrap.cc:492] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:52.354013 16807 ts_tablet_manager.cc:1367] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240502 14:14:52.355501 16808 tablet_bootstrap.cc:492] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:52.355718 16807 raft_consensus.cc:348] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.356016 16808 ts_tablet_manager.cc:1367] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240502 14:14:52.356144 16809 tablet_bootstrap.cc:492] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:52.356288 16809 ts_tablet_manager.cc:1367] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240502 14:14:52.356881 16807 raft_consensus.cc:374] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.357002 16808 raft_consensus.cc:348] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:52.357024 16809 raft_consensus.cc:348] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.357188 16809 raft_consensus.cc:374] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.357303 16809 raft_consensus.cc:729] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.357594 16809 consensus_queue.cc:260] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.358309 16808 raft_consensus.cc:374] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.358870 16809 ts_tablet_manager.cc:1398] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:52.359228 16710 tablet_bootstrap.cc:492] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:52.359462 16710 ts_tablet_manager.cc:1367] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.003s
I20240502 14:14:52.360289 16710 raft_consensus.cc:348] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:52.360569 16710 raft_consensus.cc:374] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.360707 16710 raft_consensus.cc:729] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.360996 16710 consensus_queue.cc:260] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:52.362532 16710 ts_tablet_manager.cc:1398] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:52.362864 16808 raft_consensus.cc:729] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.363164 16808 consensus_queue.cc:260] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:52.363817 16808 ts_tablet_manager.cc:1398] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:14:52.373639 16807 raft_consensus.cc:729] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.374601 16807 consensus_queue.cc:260] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.379500 15839 raft_consensus.cc:482] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:52.379668 15839 raft_consensus.cc:504] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.380533 15839 leader_election.cc:288] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:14:52.380868 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d6faae2add42c5bd4af6018a13b841" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:52.381062 13446 raft_consensus.cc:2454] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:52.381464 13514 leader_election.cc:302] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:52.381750 15839 raft_consensus.cc:2781] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:52.381848 15839 raft_consensus.cc:482] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:52.381939 15839 raft_consensus.cc:3037] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.383514 16807 ts_tablet_manager.cc:1398] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240502 14:14:52.384579 15839 raft_consensus.cc:504] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.386322 13040 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:52.386499 13040 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:52.389081 15839 leader_election.cc:288] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:14:52.389595 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d6faae2add42c5bd4af6018a13b841" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:52.389784 13446 raft_consensus.cc:3037] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.391953 13040 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet b78249b66e93406cb271fcbbc217ecea
I20240502 14:14:52.392661 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet b78249b66e93406cb271fcbbc217ecea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:52.392756 16807 tablet_bootstrap.cc:492] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:52.392926 16526 tablet_replica.cc:330] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:52.393105 16526 raft_consensus.cc:2229] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.393622 16526 raft_consensus.cc:2258] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.394273 16807 tablet_bootstrap.cc:654] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.394253 13446 raft_consensus.cc:2454] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:52.395023 16526 ts_tablet_manager.cc:1875] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.404094 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet b78249b66e93406cb271fcbbc217ecea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:14:52.394675 13514 leader_election.cc:302] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:52.407981 16204 raft_consensus.cc:2781] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:52.409500 16807 tablet_bootstrap.cc:492] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:52.410511 16807 ts_tablet_manager.cc:1367] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 14:14:52.411371 16204 raft_consensus.cc:686] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:52.411733 16204 consensus_queue.cc:237] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.412619 16807 raft_consensus.cc:348] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:52.413923 16807 raft_consensus.cc:374] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.414299 16807 raft_consensus.cc:729] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.414872 16807 consensus_queue.cc:260] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:52.419620 16807 ts_tablet_manager.cc:1398] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:14:52.420526 16831 tablet_replica.cc:330] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:14:52.420845 16831 raft_consensus.cc:2229] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.421214 16831 raft_consensus.cc:2258] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.421950 16831 ts_tablet_manager.cc:1875] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.426661 13040 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 14:14:52.426836 13040 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240502 14:14:52.429733 16552 raft_consensus.cc:482] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:52.429926 16552 raft_consensus.cc:504] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:52.430889 13040 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 3ef68f6c2e4245dd86895c93ec270392
I20240502 14:14:52.431452 13040 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet dcc6274abeda40d78644d1685fa44ca6
I20240502 14:14:52.431483 16552 leader_election.cc:288] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:52.431849 13040 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 7bdabe4a87ec4398b6e4508ccc0ae165
I20240502 14:14:52.432066 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc6274abeda40d78644d1685fa44ca6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:52.432485 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef68f6c2e4245dd86895c93ec270392 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:52.432670 16538 tablet_replica.cc:330] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:52.432888 16538 raft_consensus.cc:2229] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.433171 16538 raft_consensus.cc:2258] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.434010 16538 ts_tablet_manager.cc:1875] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.437462 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef68f6c2e4245dd86895c93ec270392 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:52.438719 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdabe4a87ec4398b6e4508ccc0ae165 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:52.438879 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc6274abeda40d78644d1685fa44ca6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:52.440781 16526 ts_tablet_manager.cc:1888] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:52.440961 16526 log.cc:1196] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/b78249b66e93406cb271fcbbc217ecea
I20240502 14:14:52.445061 16538 ts_tablet_manager.cc:1888] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.447806 16831 ts_tablet_manager.cc:1888] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:52.447989 16831 log.cc:1196] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/b78249b66e93406cb271fcbbc217ecea
I20240502 14:14:52.448659 16831 ts_tablet_manager.cc:1909] T b78249b66e93406cb271fcbbc217ecea P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:14:52.449990 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet b78249b66e93406cb271fcbbc217ecea (table rebalancer_dynamic_table_new_name [id=53e326f05b5545f58887e5f17abc4ab7]) successfully deleted
I20240502 14:14:52.454203 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4af5f41674aaa8587158f25e495fb" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:14:52.454587 13446 raft_consensus.cc:2454] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:14:52.455050 16219 leader_election.cc:302] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:52.455418 16552 raft_consensus.cc:2781] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:52.455571 16552 raft_consensus.cc:482] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:52.455669 16552 raft_consensus.cc:3037] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.460793 16552 raft_consensus.cc:504] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:52.462026 16552 leader_election.cc:288] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:52.462014 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4af5f41674aaa8587158f25e495fb" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:14:52.462252 13446 raft_consensus.cc:3037] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.464861 13446 raft_consensus.cc:2454] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:14:52.465392 16219 leader_election.cc:302] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:52.467454 16552 raft_consensus.cc:482] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:52.467816 16552 raft_consensus.cc:504] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.469624 16538 log.cc:1196] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/dcc6274abeda40d78644d1685fa44ca6
I20240502 14:14:52.470202 16538 ts_tablet_manager.cc:1909] T dcc6274abeda40d78644d1685fa44ca6 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:52.471390 16538 tablet_replica.cc:330] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:52.471707 16538 raft_consensus.cc:2229] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.471769 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet dcc6274abeda40d78644d1685fa44ca6 (table rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]) successfully deleted
I20240502 14:14:52.472045 16538 raft_consensus.cc:2258] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.472710 16538 ts_tablet_manager.cc:1875] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.474555 16526 ts_tablet_manager.cc:1909] T b78249b66e93406cb271fcbbc217ecea P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:52.475584 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet b78249b66e93406cb271fcbbc217ecea (table rebalancer_dynamic_table_new_name [id=53e326f05b5545f58887e5f17abc4ab7]) successfully deleted
I20240502 14:14:52.475937 16526 tablet_replica.cc:330] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:52.476227 16526 raft_consensus.cc:2229] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.476588 16526 raft_consensus.cc:2258] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.477432 16526 ts_tablet_manager.cc:1875] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.482113 16552 leader_election.cc:288] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:52.483093 16552 raft_consensus.cc:2781] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:52.483486 16552 raft_consensus.cc:686] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:14:52.485685 16552 consensus_queue.cc:237] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
W20240502 14:14:52.501940 13384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.194:33193, user_credentials={real_user=slave}} blocked reactor thread for 90876.7us
I20240502 14:14:52.503167 13040 catalog_manager.cc:5484] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:52.505530 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdabe4a87ec4398b6e4508ccc0ae165 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:52.505507 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:52.507344 13429 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef68f6c2e4245dd86895c93ec270392 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:52.515623 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 069962b3cc734d54b02b7bd867022ced (DEFAULT_TABLE table=rebalancer_dynamic_table [id=de9f66d95ef14fd1ad734545b316bf14]), partition=
W20240502 14:14:52.515972 13516 consensus_peers.cc:485] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcc6274abeda40d78644d1685fa44ca6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:52.516460 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069962b3cc734d54b02b7bd867022ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.517978 16401 tablet_service.cc:1467] Processing CreateTablet for tablet 069962b3cc734d54b02b7bd867022ced (DEFAULT_TABLE table=rebalancer_dynamic_table [id=de9f66d95ef14fd1ad734545b316bf14]), partition=
I20240502 14:14:52.518347 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069962b3cc734d54b02b7bd867022ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.519958 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 069962b3cc734d54b02b7bd867022ced (DEFAULT_TABLE table=rebalancer_dynamic_table [id=de9f66d95ef14fd1ad734545b316bf14]), partition=
I20240502 14:14:52.520339 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 069962b3cc734d54b02b7bd867022ced. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:52.530956 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet b78249b66e93406cb271fcbbc217ecea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:52.547147 16809 tablet_bootstrap.cc:492] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:52.548949 16809 tablet_bootstrap.cc:654] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.551755 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet dcc6274abeda40d78644d1685fa44ca6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:52.556268 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet 7bdabe4a87ec4398b6e4508ccc0ae165 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:58526
W20240502 14:14:52.557493 16219 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.196.195:46303, user_credentials={real_user=slave}} blocked reactor thread for 90447.1us
I20240502 14:14:52.558907 13166 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:14:52.582923 16844 tablet_bootstrap.cc:492] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:52.583593 16843 tablet_replica.cc:330] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:52.583968 16843 raft_consensus.cc:2229] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.587342 16809 tablet_bootstrap.cc:492] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:52.587586 16809 ts_tablet_manager.cc:1367] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.012s
I20240502 14:14:52.593298 16845 tablet_bootstrap.cc:492] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:52.595666 16843 raft_consensus.cc:2258] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.598008 16842 tablet_replica.cc:330] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:52.597865 13046 catalog_manager.cc:5484] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:52.598631 16842 raft_consensus.cc:2229] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.598922 16842 raft_consensus.cc:2258] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.599592 16842 ts_tablet_manager.cc:1875] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.602479 16843 ts_tablet_manager.cc:1875] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.602455 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d6faae2add42c5bd4af6018a13b841" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:14:52.602805 16288 raft_consensus.cc:2454] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:52.602850 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4af5f41674aaa8587158f25e495fb" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:14:52.603102 16023 raft_consensus.cc:2454] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:14:52.603370 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d6faae2add42c5bd4af6018a13b841" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:14:52.603431 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4af5f41674aaa8587158f25e495fb" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:14:52.603586 16022 raft_consensus.cc:3037] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.603585 16288 raft_consensus.cc:3037] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.604277 16844 tablet_bootstrap.cc:654] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.606099 16288 raft_consensus.cc:2454] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:52.606606 16809 raft_consensus.cc:348] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:52.606690 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da975869e82479b91fdda30680e6b3c" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:14:52.606882 16845 tablet_bootstrap.cc:654] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:52.606930 16021 raft_consensus.cc:2454] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:14:52.607486 16022 raft_consensus.cc:2454] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:14:52.607569 16809 raft_consensus.cc:374] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.607717 16809 raft_consensus.cc:729] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.608115 16809 consensus_queue.cc:260] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:52.608302 16221 leader_election.cc:302] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:52.609109 16839 raft_consensus.cc:2781] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:52.609310 16839 raft_consensus.cc:482] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:52.609537 16839 raft_consensus.cc:3037] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.610355 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da975869e82479b91fdda30680e6b3c" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:52.610532 16421 raft_consensus.cc:2454] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:14:52.612488 13560 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:14:52.613062 13427 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:14:52.614449 16809 ts_tablet_manager.cc:1398] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.027s user 0.000s sys 0.001s
I20240502 14:14:52.614823 16839 raft_consensus.cc:504] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.616195 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da975869e82479b91fdda30680e6b3c" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:14:52.616366 16421 raft_consensus.cc:3037] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.616497 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da975869e82479b91fdda30680e6b3c" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:14:52.616716 16022 raft_consensus.cc:3037] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.618880 16421 raft_consensus.cc:2454] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:14:52.619238 16220 leader_election.cc:302] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:14:52.619773 16552 raft_consensus.cc:2781] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:52.622314 16022 raft_consensus.cc:2454] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:14:52.624737 16839 leader_election.cc:288] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:52.627812 16401 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:52.630872 16538 ts_tablet_manager.cc:1888] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.631021 16538 log.cc:1196] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/3ef68f6c2e4245dd86895c93ec270392
I20240502 14:14:52.631394 16538 ts_tablet_manager.cc:1909] T 3ef68f6c2e4245dd86895c93ec270392 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:52.632366 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 3ef68f6c2e4245dd86895c93ec270392 (table rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]) successfully deleted
I20240502 14:14:52.637176 16552 raft_consensus.cc:686] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:14:52.637482 16552 consensus_queue.cc:237] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:52.638500 16844 tablet_bootstrap.cc:492] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:52.638662 16844 ts_tablet_manager.cc:1367] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.003s
I20240502 14:14:52.638957 16526 ts_tablet_manager.cc:1888] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.639142 16526 log.cc:1196] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/3ef68f6c2e4245dd86895c93ec270392
I20240502 14:14:52.639653 16526 ts_tablet_manager.cc:1909] T 3ef68f6c2e4245dd86895c93ec270392 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:52.639740 16844 raft_consensus.cc:348] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:52.639999 16844 raft_consensus.cc:374] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.640136 16844 raft_consensus.cc:729] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.640383 16844 consensus_queue.cc:260] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:52.640955 16844 ts_tablet_manager.cc:1398] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:52.642179 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 3ef68f6c2e4245dd86895c93ec270392 (table rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]) successfully deleted
I20240502 14:14:52.642763 16845 tablet_bootstrap.cc:492] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:52.642937 16845 ts_tablet_manager.cc:1367] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.000s
I20240502 14:14:52.643994 16845 raft_consensus.cc:348] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:52.644690 16526 tablet_replica.cc:330] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:52.644925 16845 raft_consensus.cc:374] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:52.645090 16845 raft_consensus.cc:729] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:52.645135 16526 raft_consensus.cc:2229] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.645467 16845 consensus_queue.cc:260] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:52.645699 16526 raft_consensus.cc:2258] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.646647 16526 ts_tablet_manager.cc:1875] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.648021 16845 ts_tablet_manager.cc:1398] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:14:52.650480 16842 ts_tablet_manager.cc:1888] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.650653 16842 log.cc:1196] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/7bdabe4a87ec4398b6e4508ccc0ae165
I20240502 14:14:52.651104 16842 ts_tablet_manager.cc:1909] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:52.652343 16842 tablet_replica.cc:330] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:52.652611 16842 raft_consensus.cc:2229] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.652853 16842 raft_consensus.cc:2258] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.653915 16842 ts_tablet_manager.cc:1875] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.659579 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 7bdabe4a87ec4398b6e4508ccc0ae165 (table rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]) successfully deleted
I20240502 14:14:52.662866 16842 ts_tablet_manager.cc:1888] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.663148 16842 log.cc:1196] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/3ef68f6c2e4245dd86895c93ec270392
I20240502 14:14:52.665089 16842 ts_tablet_manager.cc:1909] T 3ef68f6c2e4245dd86895c93ec270392 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
W20240502 14:14:52.666693 13514 consensus_peers.cc:485] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bdabe4a87ec4398b6e4508ccc0ae165. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:52.668249 16526 ts_tablet_manager.cc:1888] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.668346 16526 log.cc:1196] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/7bdabe4a87ec4398b6e4508ccc0ae165
I20240502 14:14:52.668802 16526 ts_tablet_manager.cc:1909] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:52.669379 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 3ef68f6c2e4245dd86895c93ec270392 (table rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]) successfully deleted
I20240502 14:14:52.669754 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 7bdabe4a87ec4398b6e4508ccc0ae165 (table rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]) successfully deleted
I20240502 14:14:52.671679 16526 tablet_replica.cc:330] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:52.671880 16526 raft_consensus.cc:2229] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.672080 16526 raft_consensus.cc:2258] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.673946 16526 ts_tablet_manager.cc:1875] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.681738 16526 ts_tablet_manager.cc:1888] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.685387 13046 catalog_manager.cc:5484] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:52.686399 16266 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:52.686582 16526 log.cc:1196] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/dcc6274abeda40d78644d1685fa44ca6
I20240502 14:14:52.686985 16526 ts_tablet_manager.cc:1909] T dcc6274abeda40d78644d1685fa44ca6 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:52.687804 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet dcc6274abeda40d78644d1685fa44ca6 (table rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]) successfully deleted
I20240502 14:14:52.691015 16002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:52.695477 16843 ts_tablet_manager.cc:1888] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:52.696784 16843 log.cc:1196] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/b78249b66e93406cb271fcbbc217ecea
I20240502 14:14:52.699453 16843 ts_tablet_manager.cc:1909] T b78249b66e93406cb271fcbbc217ecea P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:52.700429 16843 tablet_replica.cc:330] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:52.700522 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet b78249b66e93406cb271fcbbc217ecea (table rebalancer_dynamic_table_new_name [id=53e326f05b5545f58887e5f17abc4ab7]) successfully deleted
I20240502 14:14:52.707435 13297 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:14:52.725335 16134 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:52.726377 16843 raft_consensus.cc:2229] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.726707 16843 raft_consensus.cc:2258] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.735901 16843 ts_tablet_manager.cc:1875] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.744066 16843 ts_tablet_manager.cc:1888] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.744189 16843 log.cc:1196] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/dcc6274abeda40d78644d1685fa44ca6
I20240502 14:14:52.756933 16843 ts_tablet_manager.cc:1909] T dcc6274abeda40d78644d1685fa44ca6 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:52.757951 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet dcc6274abeda40d78644d1685fa44ca6 (table rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]) successfully deleted
I20240502 14:14:52.758613 16843 tablet_replica.cc:330] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:52.758822 16843 raft_consensus.cc:2229] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.759209 16843 raft_consensus.cc:2258] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.760035 16843 ts_tablet_manager.cc:1875] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.773213 16843 ts_tablet_manager.cc:1888] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.773399 16843 log.cc:1196] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/7bdabe4a87ec4398b6e4508ccc0ae165
I20240502 14:14:52.774960 16843 ts_tablet_manager.cc:1909] T 7bdabe4a87ec4398b6e4508ccc0ae165 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:52.780745 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 7bdabe4a87ec4398b6e4508ccc0ae165 (table rebalance_test_table_8 [id=75d4a7920a3643c5adb07f84cba064bd]) successfully deleted
I20240502 14:14:52.879396 13446 consensus_queue.cc:237] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:52.881790 13187 raft_consensus.cc:1261] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:52.882246 13579 raft_consensus.cc:1261] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:52.884164 16832 consensus_queue.cc:1035] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.884481 16338 consensus_queue.cc:1035] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:52.884744 16657 consensus_queue.cc:1035] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:14:52.895098 13384 consensus_peers.cc:485] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcf9e4934b4d4400884eee33f42b29f5. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:52.901489 16832 consensus_queue.cc:1035] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:52.904574 16657 raft_consensus.cc:2932] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:52.905053 13187 raft_consensus.cc:2932] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:52.914535 13046 catalog_manager.cc:5484] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:52.915125 16738 raft_consensus.cc:482] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:52.915287 16738 raft_consensus.cc:504] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:52.915999 16738 leader_election.cc:288] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:52.916846 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069962b3cc734d54b02b7bd867022ced" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:52.917097 16154 raft_consensus.cc:2454] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:14:52.917946 16354 leader_election.cc:302] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:52.918344 16738 raft_consensus.cc:2781] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:52.918440 16738 raft_consensus.cc:482] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:52.918551 16738 raft_consensus.cc:3037] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.922199 16738 raft_consensus.cc:504] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:52.923324 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069962b3cc734d54b02b7bd867022ced" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:52.923547 13188 raft_consensus.cc:3037] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.923777 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069962b3cc734d54b02b7bd867022ced" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:52.923940 16154 raft_consensus.cc:3037] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:52.926352 13188 raft_consensus.cc:2454] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:14:52.926384 16154 raft_consensus.cc:2454] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:14:52.927629 16353 leader_election.cc:302] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:14:52.927891 16741 raft_consensus.cc:2781] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:52.928025 16741 raft_consensus.cc:686] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:14:52.928323 16741 consensus_queue.cc:237] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:52.928809 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "069962b3cc734d54b02b7bd867022ced" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:52.928978 13187 raft_consensus.cc:2362] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1. Re-sending same reply.
I20240502 14:14:52.929262 16738 leader_election.cc:288] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:52.932339 13046 catalog_manager.cc:5484] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:52.933480 13036 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 14:14:52.933666 13036 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240502 14:14:52.936870 13036 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 373143a499c0492991c0693caf245ff4
I20240502 14:14:52.937438 13036 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet eaba1d442abf493bac8cce1c4ab9cf5c
I20240502 14:14:52.937803 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 373143a499c0492991c0693caf245ff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:52.937866 13036 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 525ac3ddd7a148e89bc3dc241e9854d0
I20240502 14:14:52.938088 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 373143a499c0492991c0693caf245ff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:52.938185 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet eaba1d442abf493bac8cce1c4ab9cf5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:52.938421 16842 tablet_replica.cc:330] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:52.938526 16843 tablet_replica.cc:330] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:52.938539 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet 373143a499c0492991c0693caf245ff4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:52.938637 16842 raft_consensus.cc:2229] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.938745 16843 raft_consensus.cc:2229] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.938843 16538 tablet_replica.cc:330] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:52.938957 16843 raft_consensus.cc:2258] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.938969 16842 raft_consensus.cc:2258] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.939126 16538 raft_consensus.cc:2229] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.939388 16538 raft_consensus.cc:2258] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.939622 16842 ts_tablet_manager.cc:1875] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.939653 16843 ts_tablet_manager.cc:1875] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.940290 16538 ts_tablet_manager.cc:1875] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.941630 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet eaba1d442abf493bac8cce1c4ab9cf5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:52.942209 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 525ac3ddd7a148e89bc3dc241e9854d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:52.945377 16526 tablet_replica.cc:330] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:52.945550 16526 raft_consensus.cc:2229] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.945830 16526 raft_consensus.cc:2258] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.946298 16842 ts_tablet_manager.cc:1888] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.946386 16842 log.cc:1196] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/eaba1d442abf493bac8cce1c4ab9cf5c
I20240502 14:14:52.946455 16526 ts_tablet_manager.cc:1875] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.946825 16842 ts_tablet_manager.cc:1909] T eaba1d442abf493bac8cce1c4ab9cf5c P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:52.947990 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet eaba1d442abf493bac8cce1c4ab9cf5c (table rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]) successfully deleted
I20240502 14:14:52.949294 16538 ts_tablet_manager.cc:1888] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.949430 16538 log.cc:1196] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/373143a499c0492991c0693caf245ff4
I20240502 14:14:52.949940 16538 ts_tablet_manager.cc:1909] T 373143a499c0492991c0693caf245ff4 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:52.950320 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet 525ac3ddd7a148e89bc3dc241e9854d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:52.950548 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet eaba1d442abf493bac8cce1c4ab9cf5c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:52.950594 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet 525ac3ddd7a148e89bc3dc241e9854d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:52 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:52.950883 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 373143a499c0492991c0693caf245ff4 (table rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]) successfully deleted
I20240502 14:14:52.951115 16538 tablet_replica.cc:330] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:52.951293 16538 raft_consensus.cc:2229] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:52.951525 16538 raft_consensus.cc:2258] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.951934 16526 ts_tablet_manager.cc:1888] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.952080 16526 log.cc:1196] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/373143a499c0492991c0693caf245ff4
I20240502 14:14:52.952123 16538 ts_tablet_manager.cc:1875] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.952478 16526 ts_tablet_manager.cc:1909] T 373143a499c0492991c0693caf245ff4 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:52.938537 13579 raft_consensus.cc:2932] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:52.953478 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 373143a499c0492991c0693caf245ff4 (table rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]) successfully deleted
I20240502 14:14:52.953733 16526 tablet_replica.cc:330] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:52.953928 16526 raft_consensus.cc:2229] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.954167 16526 raft_consensus.cc:2258] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.954841 16526 ts_tablet_manager.cc:1875] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.955189 16843 ts_tablet_manager.cc:1888] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.955328 16843 log.cc:1196] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/373143a499c0492991c0693caf245ff4
I20240502 14:14:52.955739 16843 ts_tablet_manager.cc:1909] T 373143a499c0492991c0693caf245ff4 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:52.957875 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 373143a499c0492991c0693caf245ff4 (table rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]) successfully deleted
I20240502 14:14:52.958760 16843 tablet_replica.cc:330] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:52.959023 16843 raft_consensus.cc:2229] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.959318 16843 raft_consensus.cc:2258] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.960175 16843 ts_tablet_manager.cc:1875] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.963618 16526 ts_tablet_manager.cc:1888] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.963634 16538 ts_tablet_manager.cc:1888] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.963809 16538 log.cc:1196] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/525ac3ddd7a148e89bc3dc241e9854d0
I20240502 14:14:52.963825 16526 log.cc:1196] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/eaba1d442abf493bac8cce1c4ab9cf5c
I20240502 14:14:52.964320 16526 ts_tablet_manager.cc:1909] T eaba1d442abf493bac8cce1c4ab9cf5c P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:52.964587 16538 ts_tablet_manager.cc:1909] T 525ac3ddd7a148e89bc3dc241e9854d0 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:52.965238 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet eaba1d442abf493bac8cce1c4ab9cf5c (table rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]) successfully deleted
I20240502 14:14:52.965901 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 525ac3ddd7a148e89bc3dc241e9854d0 (table rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]) successfully deleted
I20240502 14:14:52.966349 16843 ts_tablet_manager.cc:1888] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.966511 16843 log.cc:1196] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/eaba1d442abf493bac8cce1c4ab9cf5c
I20240502 14:14:52.966763 16526 tablet_replica.cc:330] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:52.966964 16526 raft_consensus.cc:2229] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.967082 16843 ts_tablet_manager.cc:1909] T eaba1d442abf493bac8cce1c4ab9cf5c P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:52.967217 16526 raft_consensus.cc:2258] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.967850 16526 ts_tablet_manager.cc:1875] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.967948 16843 tablet_replica.cc:330] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:52.968015 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet eaba1d442abf493bac8cce1c4ab9cf5c (table rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]) successfully deleted
I20240502 14:14:52.968145 16843 raft_consensus.cc:2229] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:52.968338 16843 raft_consensus.cc:2258] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:52.969008 16843 ts_tablet_manager.cc:1875] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:52.974115 16526 ts_tablet_manager.cc:1888] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.974225 16526 log.cc:1196] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/525ac3ddd7a148e89bc3dc241e9854d0
I20240502 14:14:52.974575 16526 ts_tablet_manager.cc:1909] T 525ac3ddd7a148e89bc3dc241e9854d0 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:52.974944 16843 ts_tablet_manager.cc:1888] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:52.975121 16843 log.cc:1196] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/525ac3ddd7a148e89bc3dc241e9854d0
I20240502 14:14:52.975467 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 525ac3ddd7a148e89bc3dc241e9854d0 (table rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]) successfully deleted
I20240502 14:14:52.975629 16843 ts_tablet_manager.cc:1909] T 525ac3ddd7a148e89bc3dc241e9854d0 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:52.976348 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 525ac3ddd7a148e89bc3dc241e9854d0 (table rebalance_test_table_10 [id=eed0aa41c10e41e79b79d17916bc3464]) successfully deleted
I20240502 14:14:52.977002 13315 consensus_queue.cc:237] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:52.979970 13188 raft_consensus.cc:1261] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:52.980469 13446 raft_consensus.cc:1261] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:52.980625 16881 consensus_queue.cc:1035] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:52.981247 16531 consensus_queue.cc:1035] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:52.982717 16552 consensus_queue.cc:1035] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:14:52.988562 13252 consensus_peers.cc:485] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba519bb3fb534e7fb6106f402e27446d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:52.990242 16881 raft_consensus.cc:2932] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:52.990828 13188 raft_consensus.cc:2932] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:52.996670 13036 catalog_manager.cc:5484] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:53.001256 13446 raft_consensus.cc:2932] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:53.014290 16552 consensus_queue.cc:1035] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.028075 13315 consensus_queue.cc:237] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:53.030862 13579 raft_consensus.cc:1261] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.030910 13188 raft_consensus.cc:1261] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.031442 16872 consensus_queue.cc:1035] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.031740 16531 consensus_queue.cc:1035] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.034051 16872 raft_consensus.cc:2932] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
W20240502 14:14:53.036000 13252 consensus_peers.cc:485] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 718223dd8fd3432cb174a53325dc784c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:53.036329 13579 raft_consensus.cc:2932] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:53.037380 13188 raft_consensus.cc:2932] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:53.040449 13046 catalog_manager.cc:5484] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:53.057123 13579 consensus_queue.cc:237] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:53.059351 13315 raft_consensus.cc:1261] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.059351 13188 raft_consensus.cc:1261] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:53.059630 13516 consensus_peers.cc:485] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bbd12f1c9884497b82c941b7231645bb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:53.060011 15839 consensus_queue.cc:1035] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.060335 16832 consensus_queue.cc:1035] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.062961 16552 consensus_queue.cc:1035] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.063350 16832 raft_consensus.cc:2932] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:53.067433 16851 consensus_queue.cc:1035] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.077440 13188 raft_consensus.cc:2932] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:53.081336 13036 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:53.090245 13044 catalog_manager.cc:5484] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:53.094350 16154 raft_consensus.cc:1261] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:53.095047 16741 consensus_queue.cc:1035] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.094920 13319 raft_consensus.cc:2932] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:53.102314 13188 raft_consensus.cc:1261] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:53.102981 16741 consensus_queue.cc:1035] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.130533 16906 mvcc.cc:205] Tried to move back new op lower bound from 7023244464506789888 to 7023244463837319168. Current Snapshot: MvccSnapshot[applied={T|T < 7023244464506789888}]
I20240502 14:14:53.132498 13036 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet 069962b3cc734d54b02b7bd867022ced (table rebalancer_dynamic_table [id=de9f66d95ef14fd1ad734545b316bf14]). Expected version 1 got 0
I20240502 14:14:53.132795 16909 mvcc.cc:205] Tried to move back new op lower bound from 7023244464506789888 to 7023244463837319168. Current Snapshot: MvccSnapshot[applied={T|T < 7023244464506789888}]
I20240502 14:14:53.134052 13046 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 069962b3cc734d54b02b7bd867022ced (table rebalancer_dynamic_table [id=de9f66d95ef14fd1ad734545b316bf14]). Expected version 1 got 0
I20240502 14:14:53.141489 16917 tablet.cc:1722] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:53.142345 16917 tablet.cc:1620] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:14:53.150336 13046 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=de9f66d95ef14fd1ad734545b316bf14] alter complete (version 1)
I20240502 14:14:53.158493 16421 consensus_queue.cc:237] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:14:53.159076 16919 tablet.cc:1722] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:53.159590 13046 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 069962b3cc734d54b02b7bd867022ced (table rebalancer_dynamic_table [id=de9f66d95ef14fd1ad734545b316bf14]). Expected version 1 got 0
I20240502 14:14:53.159611 16919 tablet.cc:1620] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:14:53.157552 16918 tablet.cc:1722] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:53.161353 16918 tablet.cc:1620] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:53.163112 16288 raft_consensus.cc:1261] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:53.163832 16741 consensus_queue.cc:1035] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:53.164916 13046 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet 069962b3cc734d54b02b7bd867022ced (table rebalancer_dynamic_table [id=de9f66d95ef14fd1ad734545b316bf14]). Expected version 1 got 0
I20240502 14:14:53.166738 16738 raft_consensus.cc:2932] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:14:53.169899 16288 raft_consensus.cc:2932] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:14:53.175827 13188 raft_consensus.cc:1261] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:14:53.176674 16353 consensus_peers.cc:485] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ab40fe08f714c99a5961cc2ad0915d3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:53.177037 16741 consensus_queue.cc:1035] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:53.179746 13188 raft_consensus.cc:2932] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:14:53.184712 13046 catalog_manager.cc:5484] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:14:53.202280 13319 consensus_queue.cc:237] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:53.204437 13581 raft_consensus.cc:1261] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:53.204474 16021 raft_consensus.cc:1261] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:14:53.204771 13252 consensus_peers.cc:485] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eab73d39b1f7455bba7aef5e515805c7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:53.204964 16872 consensus_queue.cc:1035] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:53.205173 16531 consensus_queue.cc:1035] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:53.207557 16872 raft_consensus.cc:2932] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:53.208106 13581 raft_consensus.cc:2932] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:53.211134 16021 raft_consensus.cc:2932] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:53.214655 13046 catalog_manager.cc:5484] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:53.233001 13188 consensus_queue.cc:237] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:53.235366 13581 raft_consensus.cc:1261] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.235570 13319 raft_consensus.cc:1261] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.235879 16754 consensus_queue.cc:1035] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.236182 16937 consensus_queue.cc:1035] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.241336 15881 raft_consensus.cc:2932] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:53.244783 13581 raft_consensus.cc:2932] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
W20240502 14:14:53.245193 13121 consensus_peers.cc:485] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f966f36415994f9eaa8046b407135761. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:53.246332 13044 catalog_manager.cc:5484] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:53.249183 13319 raft_consensus.cc:2932] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:53.259254 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:53.262676 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:53.268505 13446 consensus_queue.cc:237] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:53.270977 16917 tablet.cc:1722] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:53.271548 16917 tablet.cc:1620] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:14:53.272480 16134 tablet_service.cc:1467] Processing CreateTablet for tablet ba8299c5f722440e92803100e7240723 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.272854 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8299c5f722440e92803100e7240723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.272851 16401 tablet_service.cc:1467] Processing CreateTablet for tablet f9c268e5a8e7462ab8e940775c73e591 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.273283 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c268e5a8e7462ab8e940775c73e591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.277596 16399 tablet_service.cc:1467] Processing CreateTablet for tablet f5bf499e8c044a1886989a90c6bb69fa (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.277923 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bf499e8c044a1886989a90c6bb69fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.278555 13168 tablet_service.cc:1467] Processing CreateTablet for tablet f9c268e5a8e7462ab8e940775c73e591 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.278657 16918 tablet.cc:1722] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:53.278985 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c268e5a8e7462ab8e940775c73e591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.279071 16918 tablet.cc:1620] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:53.280128 16948 tablet_bootstrap.cc:492] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:53.281961 13561 tablet_service.cc:1467] Processing CreateTablet for tablet f5bf499e8c044a1886989a90c6bb69fa (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.282480 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bf499e8c044a1886989a90c6bb69fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.282671 13559 tablet_service.cc:1467] Processing CreateTablet for tablet f9c268e5a8e7462ab8e940775c73e591 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.283035 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c268e5a8e7462ab8e940775c73e591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.285554 13560 tablet_service.cc:1467] Processing CreateTablet for tablet ba8299c5f722440e92803100e7240723 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.285902 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8299c5f722440e92803100e7240723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.289407 16948 tablet_bootstrap.cc:654] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.289688 16919 tablet.cc:1722] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:53.290150 16919 tablet.cc:1620] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:14:53.293730 16003 tablet_service.cc:1467] Processing CreateTablet for tablet f5bf499e8c044a1886989a90c6bb69fa (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.294137 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bf499e8c044a1886989a90c6bb69fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.295297 13046 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=de9f66d95ef14fd1ad734545b316bf14] alter complete (version 2)
I20240502 14:14:53.299239 16002 tablet_service.cc:1467] Processing CreateTablet for tablet ba8299c5f722440e92803100e7240723 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:53.299621 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8299c5f722440e92803100e7240723. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:14:53.303452 13384 consensus_peers.cc:485] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0ec7ae12c17247e9b8c03659435bd4e2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:53.303653 13315 raft_consensus.cc:1261] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.303856 13581 raft_consensus.cc:1261] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.304239 16833 consensus_queue.cc:1035] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.304922 16657 consensus_queue.cc:1035] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.350402 16657 raft_consensus.cc:2932] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:53.357827 13319 raft_consensus.cc:2932] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:53.358070 13046 catalog_manager.cc:5484] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:53.358515 16955 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:53.362066 16955 tablet_bootstrap.cc:654] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.363230 16949 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:53.366875 16949 tablet_bootstrap.cc:654] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.369076 16946 ts_tablet_manager.cc:902] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:53.368585 13581 raft_consensus.cc:2932] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:53.369746 16951 tablet_bootstrap.cc:492] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:53.371969 16951 tablet_bootstrap.cc:654] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.383550 16951 tablet_bootstrap.cc:492] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:53.383803 16951 ts_tablet_manager.cc:1367] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 14:14:53.384495 16948 tablet_bootstrap.cc:492] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:53.384716 16948 ts_tablet_manager.cc:1367] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.105s user 0.001s sys 0.005s
I20240502 14:14:53.384645 16951 raft_consensus.cc:348] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:53.384912 16951 raft_consensus.cc:374] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.385044 16951 raft_consensus.cc:729] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.385366 16951 consensus_queue.cc:260] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:53.385643 16948 raft_consensus.cc:348] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:53.386015 16950 tablet_bootstrap.cc:492] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:53.386142 16948 raft_consensus.cc:374] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.386276 16948 raft_consensus.cc:729] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.387355 16951 ts_tablet_manager.cc:1398] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:53.388083 16950 tablet_bootstrap.cc:654] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.388381 16948 consensus_queue.cc:260] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:53.390475 16949 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:53.390460 16948 ts_tablet_manager.cc:1398] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:14:53.390724 16948 tablet_bootstrap.cc:492] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:53.390934 16949 ts_tablet_manager.cc:1367] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.004s
I20240502 14:14:53.393615 16949 raft_consensus.cc:348] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:53.395354 16948 tablet_bootstrap.cc:654] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.396775 16949 raft_consensus.cc:374] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.397229 16949 raft_consensus.cc:729] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.398027 16949 consensus_queue.cc:260] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:53.400337 16949 ts_tablet_manager.cc:1398] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 14:14:53.403028 13046 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:53.409427 16754 raft_consensus.cc:482] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:53.409600 16754 raft_consensus.cc:504] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:53.410357 16754 leader_election.cc:288] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:53.411743 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c268e5a8e7462ab8e940775c73e591" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:53.411968 16420 raft_consensus.cc:2454] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:14:53.412150 16955 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:53.412281 16955 ts_tablet_manager.cc:1367] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.054s user 0.000s sys 0.012s
I20240502 14:14:53.412467 16958 ts_tablet_manager.cc:902] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:53.413017 16955 raft_consensus.cc:348] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:53.413183 16955 raft_consensus.cc:374] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.413282 16955 raft_consensus.cc:729] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.413518 16955 consensus_queue.cc:260] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:53.414520 13121 leader_election.cc:302] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:14:53.414920 16955 ts_tablet_manager.cc:1398] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:53.415227 16955 tablet_bootstrap.cc:492] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:53.415711 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c268e5a8e7462ab8e940775c73e591" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:53.416522 16917 tablet.cc:1722] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:53.416576 16946 tablet_copy_client.cc:308] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
W20240502 14:14:53.417639 13120 leader_election.cc:341] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:53.422708 16918 tablet.cc:1722] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:53.427119 16955 tablet_bootstrap.cc:654] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.429636 16754 raft_consensus.cc:2781] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:53.429759 16754 raft_consensus.cc:482] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:53.429948 16754 raft_consensus.cc:3037] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.432466 16754 raft_consensus.cc:504] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:53.433166 16948 tablet_bootstrap.cc:492] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:53.433338 16948 ts_tablet_manager.cc:1367] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.001s
I20240502 14:14:53.434195 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c268e5a8e7462ab8e940775c73e591" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:14:53.434377 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=de9f66d95ef14fd1ad734545b316bf14] alter complete (version 3)
I20240502 14:14:53.436079 16948 raft_consensus.cc:348] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:53.436281 16948 raft_consensus.cc:374] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240502 14:14:53.436336 13120 leader_election.cc:341] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:53.437022 16919 tablet.cc:1722] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:53.436370 16948 raft_consensus.cc:729] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.437459 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet dcf9e4934b4d4400884eee33f42b29f5 from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:53727)
I20240502 14:14:53.437741 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet dcf9e4934b4d4400884eee33f42b29f5 from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:53727: session id = d1da7f11f0604f78934354a992bb0f55-dcf9e4934b4d4400884eee33f42b29f5
I20240502 14:14:53.437714 16948 consensus_queue.cc:260] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:53.438460 16948 ts_tablet_manager.cc:1398] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:14:53.439850 16950 tablet_bootstrap.cc:492] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:53.440012 16950 ts_tablet_manager.cc:1367] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.003s
I20240502 14:14:53.440879 16950 raft_consensus.cc:348] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:53.441052 16950 raft_consensus.cc:374] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.441155 16950 raft_consensus.cc:729] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.441351 16950 consensus_queue.cc:260] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:53.441869 16950 ts_tablet_manager.cc:1398] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:53.442162 16950 tablet_bootstrap.cc:492] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:53.442215 16955 tablet_bootstrap.cc:492] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:53.442339 16955 ts_tablet_manager.cc:1367] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.002s
I20240502 14:14:53.443135 16955 raft_consensus.cc:348] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:53.443332 16955 raft_consensus.cc:374] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.443396 16955 raft_consensus.cc:729] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.443631 16955 consensus_queue.cc:260] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:53.444664 16950 tablet_bootstrap.cc:654] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.444968 16955 ts_tablet_manager.cc:1398] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:14:53.446192 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c268e5a8e7462ab8e940775c73e591" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:14:53.446357 16421 raft_consensus.cc:3037] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.448880 16421 raft_consensus.cc:2454] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:14:53.449415 13470 tablet_copy_source_session.cc:205] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:53.450525 13121 leader_election.cc:302] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968; no voters: 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:14:53.450588 13044 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 14:14:53.450738 13044 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240502 14:14:53.450803 15881 raft_consensus.cc:2781] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:53.450915 15881 raft_consensus.cc:686] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:14:53.451149 15881 consensus_queue.cc:237] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:53.452199 16754 leader_election.cc:288] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:53.456534 13044 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet d7b9a8e040f341ca8e25727310994df0
I20240502 14:14:53.457140 16804 raft_consensus.cc:482] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:53.457310 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet d7b9a8e040f341ca8e25727310994df0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:53.457340 16804 raft_consensus.cc:504] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:53.458328 16804 leader_election.cc:288] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:53.458639 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet d7b9a8e040f341ca8e25727310994df0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:53.458935 16538 tablet_replica.cc:330] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:53.459156 16538 raft_consensus.cc:2229] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.459417 16538 raft_consensus.cc:2258] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.459379 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8299c5f722440e92803100e7240723" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:14:53.459569 16021 raft_consensus.cc:2454] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:14:53.460029 16090 leader_election.cc:302] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:53.460155 16538 ts_tablet_manager.cc:1875] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.460300 16804 raft_consensus.cc:2781] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:53.460395 16804 raft_consensus.cc:482] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:53.460474 16804 raft_consensus.cc:3037] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.461504 16946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcf9e4934b4d4400884eee33f42b29f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.462929 16804 raft_consensus.cc:504] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:53.463830 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8299c5f722440e92803100e7240723" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:14:53.464028 16021 raft_consensus.cc:3037] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.466487 16021 raft_consensus.cc:2454] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:14:53.467020 16090 leader_election.cc:302] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:53.467976 16804 leader_election.cc:288] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:53.468147 16804 raft_consensus.cc:2781] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:53.468253 16804 raft_consensus.cc:686] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:14:53.468492 16804 consensus_queue.cc:237] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:53.468891 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet d7b9a8e040f341ca8e25727310994df0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:53.469170 16526 tablet_replica.cc:330] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:53.469417 16526 raft_consensus.cc:2229] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.469646 16526 raft_consensus.cc:2258] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.470012 16538 ts_tablet_manager.cc:1888] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:53.470149 16538 log.cc:1196] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/d7b9a8e040f341ca8e25727310994df0
I20240502 14:14:53.470381 16526 ts_tablet_manager.cc:1875] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.470584 16538 ts_tablet_manager.cc:1909] T d7b9a8e040f341ca8e25727310994df0 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:53.476491 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet d7b9a8e040f341ca8e25727310994df0 (table rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]) successfully deleted
I20240502 14:14:53.476778 13044 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 4 replicas of tablet f966f36415994f9eaa8046b407135761
I20240502 14:14:53.477826 13561 tablet_service.cc:1514] Processing DeleteTablet for tablet f966f36415994f9eaa8046b407135761 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:53.478003 16975 tablet_replica.cc:330] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:53.478108 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet f966f36415994f9eaa8046b407135761 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:53.478227 16975 raft_consensus.cc:2229] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:53.478571 16975 raft_consensus.cc:2258] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.478597 13044 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 4 replicas of tablet dcf9e4934b4d4400884eee33f42b29f5
I20240502 14:14:53.479128 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet f966f36415994f9eaa8046b407135761 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:53.479300 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet dcf9e4934b4d4400884eee33f42b29f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:53.480087 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet dcf9e4934b4d4400884eee33f42b29f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:53.480876 16526 ts_tablet_manager.cc:1888] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:53.481004 16526 log.cc:1196] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/d7b9a8e040f341ca8e25727310994df0
I20240502 14:14:53.480939 16002 tablet_service.cc:1514] Processing DeleteTablet for tablet dcf9e4934b4d4400884eee33f42b29f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:41222
I20240502 14:14:53.482189 16946 tablet_copy_client.cc:783] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:53.482229 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet f966f36415994f9eaa8046b407135761 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:14:53.482534 16946 tablet_copy_client.cc:647] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:53.482982 16526 ts_tablet_manager.cc:1909] T d7b9a8e040f341ca8e25727310994df0 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:53.486372 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet d7b9a8e040f341ca8e25727310994df0 (table rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]) successfully deleted
I20240502 14:14:53.489573 16946 tablet_copy_client.cc:523] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:53.491137 16538 tablet_replica.cc:330] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:53.491328 16538 raft_consensus.cc:2229] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.491647 16538 raft_consensus.cc:2258] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.480202 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet dcf9e4934b4d4400884eee33f42b29f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:53.494939 16975 ts_tablet_manager.cc:1875] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.495663 16946 tablet_bootstrap.cc:492] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:53.497424 16526 tablet_replica.cc:330] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:53.497900 16526 raft_consensus.cc:2229] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:53.498167 16538 ts_tablet_manager.cc:1875] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.499096 16526 raft_consensus.cc:2258] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240502 14:14:53.500941 13030 catalog_manager.cc:4797] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): delete failed for tablet f966f36415994f9eaa8046b407135761 because the tablet was not found. No further retry: Not found: Tablet not found: f966f36415994f9eaa8046b407135761
I20240502 14:14:53.501739 16526 ts_tablet_manager.cc:1875] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.503720 16979 tablet_replica.cc:330] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
W20240502 14:14:53.509020 13031 catalog_manager.cc:4810] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): delete failed for tablet dcf9e4934b4d4400884eee33f42b29f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcf9e4934b4d4400884eee33f42b29f5 already in progress: copying tablet
I20240502 14:14:53.509542 16979 raft_consensus.cc:2229] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.511726 16979 raft_consensus.cc:2258] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.512850 16979 ts_tablet_manager.cc:1875] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.515158 13044 catalog_manager.cc:5484] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
W20240502 14:14:53.520347 16090 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.196.197:43695, user_credentials={real_user=slave}} blocked reactor thread for 53300.9us
I20240502 14:14:53.521948 13044 catalog_manager.cc:5484] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:53.523021 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8299c5f722440e92803100e7240723" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:53.523128 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8299c5f722440e92803100e7240723" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
W20240502 14:14:53.524224 16088 leader_election.cc:341] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:14:53.524853 16088 leader_election.cc:341] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:53.530093 16538 ts_tablet_manager.cc:1888] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.531293 16975 ts_tablet_manager.cc:1888] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:53.531431 16526 ts_tablet_manager.cc:1888] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.532171 16985 ts_tablet_manager.cc:902] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:53.534884 16975 log.cc:1196] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/d7b9a8e040f341ca8e25727310994df0
I20240502 14:14:53.535482 16975 ts_tablet_manager.cc:1909] T d7b9a8e040f341ca8e25727310994df0 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:53.536726 16975 tablet_replica.cc:330] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:53.536837 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet d7b9a8e040f341ca8e25727310994df0 (table rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]) successfully deleted
I20240502 14:14:53.536798 13036 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:53.537124 16975 raft_consensus.cc:2229] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:53.537129 13036 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:53.537606 16975 raft_consensus.cc:2258] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.538800 16975 ts_tablet_manager.cc:1875] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.539525 16538 log.cc:1196] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/f966f36415994f9eaa8046b407135761
I20240502 14:14:53.540124 16950 tablet_bootstrap.cc:492] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:53.539005 13446 consensus_queue.cc:237] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:53.541198 13036 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 069962b3cc734d54b02b7bd867022ced
I20240502 14:14:53.542142 16399 tablet_service.cc:1514] Processing DeleteTablet for tablet 069962b3cc734d54b02b7bd867022ced with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:53312
I20240502 14:14:53.542220 16538 ts_tablet_manager.cc:1909] T f966f36415994f9eaa8046b407135761 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:53.540943 16950 ts_tablet_manager.cc:1367] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.099s user 0.008s sys 0.004s
I20240502 14:14:53.542501 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet 069962b3cc734d54b02b7bd867022ced with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:14:53.543004 16981 tablet_replica.cc:330] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:14:53.543246 16981 raft_consensus.cc:2229] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.543184 16950 raft_consensus.cc:348] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:53.543330 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet f966f36415994f9eaa8046b407135761 (table rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]) successfully deleted
I20240502 14:14:53.543391 16950 raft_consensus.cc:374] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.543460 16950 raft_consensus.cc:729] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.543664 16950 consensus_queue.cc:260] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:53.543521 16981 raft_consensus.cc:2258] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.544168 16950 ts_tablet_manager.cc:1398] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:53.546288 16979 ts_tablet_manager.cc:1888] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.546561 16979 log.cc:1196] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/f966f36415994f9eaa8046b407135761
I20240502 14:14:53.546784 16981 ts_tablet_manager.cc:1875] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.546986 16979 ts_tablet_manager.cc:1909] T f966f36415994f9eaa8046b407135761 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:53.552448 16950 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:53.553329 16958 tablet_copy_client.cc:308] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:14:53.554046 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet f966f36415994f9eaa8046b407135761 (table rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]) successfully deleted
I20240502 14:14:53.554450 16979 tablet_replica.cc:330] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:53.556488 16979 raft_consensus.cc:2229] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.556756 16979 raft_consensus.cc:2258] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.557708 16979 ts_tablet_manager.cc:1875] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.542459 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 069962b3cc734d54b02b7bd867022ced with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:53.562157 16526 log.cc:1196] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/f966f36415994f9eaa8046b407135761
I20240502 14:14:53.563046 16950 tablet_bootstrap.cc:654] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.564675 16979 ts_tablet_manager.cc:1888] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.564806 15839 raft_consensus.cc:482] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:53.564860 16979 log.cc:1196] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/dcf9e4934b4d4400884eee33f42b29f5
I20240502 14:14:53.564942 15839 raft_consensus.cc:504] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:53.562875 16526 ts_tablet_manager.cc:1909] T f966f36415994f9eaa8046b407135761 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:53.565331 16979 ts_tablet_manager.cc:1909] T dcf9e4934b4d4400884eee33f42b29f5 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:53.565580 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet ba519bb3fb534e7fb6106f402e27446d from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:56671)
I20240502 14:14:53.565752 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet ba519bb3fb534e7fb6106f402e27446d from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:56671: session id = 8b47d4390f8343a493d6c00123785968-ba519bb3fb534e7fb6106f402e27446d
I20240502 14:14:53.565876 15839 leader_election.cc:288] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:53.566509 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c268e5a8e7462ab8e940775c73e591" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:53.567353 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet f966f36415994f9eaa8046b407135761 (table rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]) successfully deleted
I20240502 14:14:53.547919 16987 ts_tablet_manager.cc:902] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:53.568579 16526 tablet_replica.cc:330] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:53.569712 16950 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:53.570261 16526 raft_consensus.cc:2229] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.570297 16981 ts_tablet_manager.cc:1888] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.570416 16981 log.cc:1196] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/069962b3cc734d54b02b7bd867022ced
I20240502 14:14:53.570423 13339 tablet_copy_source_session.cc:205] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:53.570827 16981 ts_tablet_manager.cc:1909] T 069962b3cc734d54b02b7bd867022ced P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:14:53.571139 16526 raft_consensus.cc:2258] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.571368 16950 ts_tablet_manager.cc:1367] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240502 14:14:53.572114 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 069962b3cc734d54b02b7bd867022ced (table rebalancer_dynamic_table_new_name [id=de9f66d95ef14fd1ad734545b316bf14]) successfully deleted
I20240502 14:14:53.574707 16958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba519bb3fb534e7fb6106f402e27446d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.577775 16950 raft_consensus.cc:348] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:53.578492 16526 ts_tablet_manager.cc:1875] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.578958 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet dcf9e4934b4d4400884eee33f42b29f5 (table rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]) successfully deleted
I20240502 14:14:53.579255 16992 tablet_replica.cc:330] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:14:53.579576 16992 raft_consensus.cc:2229] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:53.580467 16992 raft_consensus.cc:2258] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.582167 16950 raft_consensus.cc:374] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.582629 16950 raft_consensus.cc:729] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.583132 16992 ts_tablet_manager.cc:1875] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.583082 16950 consensus_queue.cc:260] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:53.585654 16950 ts_tablet_manager.cc:1398] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240502 14:14:53.587726 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9c268e5a8e7462ab8e940775c73e591" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:53.588042 16421 raft_consensus.cc:2379] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7eb4ecb2379b44c0954f2658c4a04350 in current term 1: Already voted for candidate 20a967bad5504b1e9c6fb720ecdd1625 in this term.
I20240502 14:14:53.588492 13515 leader_election.cc:302] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968
I20240502 14:14:53.589237 15839 raft_consensus.cc:3037] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.592077 16958 tablet_copy_client.cc:783] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:53.592559 16958 tablet_copy_client.cc:647] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:53.592639 15839 raft_consensus.cc:2726] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:53.595607 16958 tablet_copy_client.cc:523] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:53.596932 13187 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.601948 16657 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:14:53.605681 13383 consensus_peers.cc:485] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3be42ac139741d19dd872ab551bebae. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:53.608925 16958 tablet_bootstrap.cc:492] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:53.609411 13581 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.609938 16657 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.615293 16526 ts_tablet_manager.cc:1888] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.616194 16992 ts_tablet_manager.cc:1888] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.616319 16992 log.cc:1196] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/069962b3cc734d54b02b7bd867022ced
I20240502 14:14:53.616822 16992 ts_tablet_manager.cc:1909] T 069962b3cc734d54b02b7bd867022ced P 8b47d4390f8343a493d6c00123785968: Deleting consensus metadata
I20240502 14:14:53.617939 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 069962b3cc734d54b02b7bd867022ced (table rebalancer_dynamic_table_new_name [id=de9f66d95ef14fd1ad734545b316bf14]) successfully deleted
I20240502 14:14:53.618327 16526 log.cc:1196] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/dcf9e4934b4d4400884eee33f42b29f5
I20240502 14:14:53.619011 16526 ts_tablet_manager.cc:1909] T dcf9e4934b4d4400884eee33f42b29f5 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:53.619349 16338 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:53.620043 13187 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:53.621721 16975 ts_tablet_manager.cc:1888] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.621881 16975 log.cc:1196] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/dcf9e4934b4d4400884eee33f42b29f5
I20240502 14:14:53.622305 16975 ts_tablet_manager.cc:1909] T dcf9e4934b4d4400884eee33f42b29f5 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:53.622921 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet dcf9e4934b4d4400884eee33f42b29f5 (table rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]) successfully deleted
I20240502 14:14:53.627661 13036 catalog_manager.cc:5484] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:53.629990 16987 tablet_copy_client.cc:308] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:14:53.633507 13581 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:53.644934 13044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:53.647665 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 718223dd8fd3432cb174a53325dc784c from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:40385)
I20240502 14:14:53.647842 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 718223dd8fd3432cb174a53325dc784c from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:40385: session id = c86f7440570f4796babfd44c3d7fd847-718223dd8fd3432cb174a53325dc784c
I20240502 14:14:53.651257 13339 tablet_copy_source_session.cc:205] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:53.654421 16987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 718223dd8fd3432cb174a53325dc784c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.657837 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 42dff473ffa2439580897011db7f8d38 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=89514fc00a3a4fa194eac77c3bcccedb]), partition=
I20240502 14:14:53.657881 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 42dff473ffa2439580897011db7f8d38 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=89514fc00a3a4fa194eac77c3bcccedb]), partition=
I20240502 14:14:53.657881 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 42dff473ffa2439580897011db7f8d38 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=89514fc00a3a4fa194eac77c3bcccedb]), partition=
I20240502 14:14:53.658133 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42dff473ffa2439580897011db7f8d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.658202 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42dff473ffa2439580897011db7f8d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.658197 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42dff473ffa2439580897011db7f8d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.666647 16951 tablet_bootstrap.cc:492] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:53.666653 16985 tablet_copy_client.cc:308] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:14:53.668229 16951 tablet_bootstrap.cc:654] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.671469 16949 tablet_bootstrap.cc:492] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:53.672837 16949 tablet_bootstrap.cc:654] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.673421 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet dcf9e4934b4d4400884eee33f42b29f5 (table rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]) successfully deleted
I20240502 14:14:53.675529 16526 tablet_replica.cc:330] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:53.676160 16526 raft_consensus.cc:2229] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.676913 16526 raft_consensus.cc:2258] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.678841 16526 ts_tablet_manager.cc:1875] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.686432 17005 ts_tablet_manager.cc:902] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:53.687587 17009 ts_tablet_manager.cc:902] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3: Initiating tablet copy from peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:53.696624 16526 ts_tablet_manager.cc:1888] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.697691 17010 tablet_bootstrap.cc:492] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:53.701925 16949 tablet_bootstrap.cc:492] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:53.702694 16949 ts_tablet_manager.cc:1367] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.006s
I20240502 14:14:53.704124 16949 raft_consensus.cc:348] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:53.704386 16949 raft_consensus.cc:374] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.704483 16949 raft_consensus.cc:729] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.704741 16949 consensus_queue.cc:260] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:53.704946 17010 tablet_bootstrap.cc:654] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:53.705372 16949 ts_tablet_manager.cc:1398] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:53.707412 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet bbd12f1c9884497b82c941b7231645bb from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:53105)
I20240502 14:14:53.707685 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet bbd12f1c9884497b82c941b7231645bb from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:53105: session id = 5297e816f782430d9a01220f0471a882-bbd12f1c9884497b82c941b7231645bb
I20240502 14:14:53.711321 13601 tablet_copy_source_session.cc:205] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:53.711704 16987 tablet_copy_client.cc:783] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:53.712095 16987 tablet_copy_client.cc:647] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:53.713563 16985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbd12f1c9884497b82c941b7231645bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.718953 16946 tablet_bootstrap.cc:492] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:53.703430 16951 tablet_bootstrap.cc:492] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:53.726732 16951 ts_tablet_manager.cc:1367] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.006s
I20240502 14:14:53.727010 16946 tablet_bootstrap.cc:492] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:14:53.727438 16946 ts_tablet_manager.cc:1367] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.232s user 0.052s sys 0.019s
I20240502 14:14:53.727543 16951 raft_consensus.cc:348] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:53.727761 16951 raft_consensus.cc:374] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.727841 16951 raft_consensus.cc:729] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.728071 16951 consensus_queue.cc:260] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:53.728296 16946 raft_consensus.cc:348] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:53.728554 16946 raft_consensus.cc:729] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:14:53.728682 16951 ts_tablet_manager.cc:1398] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:53.728758 16946 consensus_queue.cc:260] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:53.729060 16526 log.cc:1196] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/069962b3cc734d54b02b7bd867022ced
I20240502 14:14:53.732546 16985 tablet_copy_client.cc:783] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:53.732803 16985 tablet_copy_client.cc:647] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:53.733407 17010 tablet_bootstrap.cc:492] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:53.734017 17010 ts_tablet_manager.cc:1367] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240502 14:14:53.733188 16526 ts_tablet_manager.cc:1909] T 069962b3cc734d54b02b7bd867022ced P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:53.735306 16002 tablet_service.cc:1514] Processing DeleteTablet for tablet dcf9e4934b4d4400884eee33f42b29f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:41222
I20240502 14:14:53.735615 16985 tablet_copy_client.cc:523] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:53.735697 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 069962b3cc734d54b02b7bd867022ced (table rebalancer_dynamic_table_new_name [id=de9f66d95ef14fd1ad734545b316bf14]) successfully deleted
W20240502 14:14:53.735967 13031 catalog_manager.cc:4810] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): delete failed for tablet dcf9e4934b4d4400884eee33f42b29f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dcf9e4934b4d4400884eee33f42b29f5 already in progress: copying tablet
I20240502 14:14:53.736537 16946 ts_tablet_manager.cc:1398] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:14:53.739187 16002 tablet_service.cc:1514] Processing DeleteTablet for tablet dcf9e4934b4d4400884eee33f42b29f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:41222
I20240502 14:14:53.739518 16980 tablet_replica.cc:330] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:14:53.739995 16980 raft_consensus.cc:2229] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:14:53.740334 16980 raft_consensus.cc:2258] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:14:53.740963 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-dcf9e4934b4d4400884eee33f42b29f5 received from {username='slave'} at 127.6.196.197:53727
I20240502 14:14:53.741145 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-dcf9e4934b4d4400884eee33f42b29f5 on tablet dcf9e4934b4d4400884eee33f42b29f5 with peer d1da7f11f0604f78934354a992bb0f55
I20240502 14:14:53.741549 16980 ts_tablet_manager.cc:1875] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.742849 16987 tablet_copy_client.cc:523] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:53.743394 17010 raft_consensus.cc:348] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:53.748020 16980 ts_tablet_manager.cc:1888] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:53.748143 16980 log.cc:1196] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/dcf9e4934b4d4400884eee33f42b29f5
I20240502 14:14:53.748536 16980 ts_tablet_manager.cc:1909] T dcf9e4934b4d4400884eee33f42b29f5 P d1da7f11f0604f78934354a992bb0f55: Deleting consensus metadata
I20240502 14:14:53.748692 17010 raft_consensus.cc:374] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:53.749094 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet dcf9e4934b4d4400884eee33f42b29f5 (table rebalance_test_table_12 [id=fe3346248b4546db98fb1fc546016280]) successfully deleted
I20240502 14:14:53.749141 17010 raft_consensus.cc:729] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:53.749471 15839 raft_consensus.cc:482] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:53.749641 15839 raft_consensus.cc:504] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:53.750365 15839 leader_election.cc:288] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:53.750862 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bf499e8c044a1886989a90c6bb69fa" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:53.751112 16421 raft_consensus.cc:2454] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:53.751535 17010 consensus_queue.cc:260] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:53.752431 17025 ts_tablet_manager.cc:902] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:53.752540 13515 leader_election.cc:302] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:14:53.752777 15839 raft_consensus.cc:2781] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:53.752851 15839 raft_consensus.cc:482] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:53.752930 15839 raft_consensus.cc:3037] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.755594 15839 raft_consensus.cc:504] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:53.756134 15839 leader_election.cc:288] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:53.756184 17010 ts_tablet_manager.cc:1398] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240502 14:14:53.757073 17009 tablet_copy_client.cc:308] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.200:43857
I20240502 14:14:53.757558 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bf499e8c044a1886989a90c6bb69fa" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:14:53.757778 16421 raft_consensus.cc:3037] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.758302 16442 tablet_copy_service.cc:140] P 8b47d4390f8343a493d6c00123785968: Received BeginTabletCopySession request for tablet 3ab40fe08f714c99a5961cc2ad0915d3 from peer 88b77f6edc1c4a72b548bf47c05704f3 ({username='slave'} at 127.6.196.195:39073)
I20240502 14:14:53.758561 16442 tablet_copy_service.cc:161] P 8b47d4390f8343a493d6c00123785968: Beginning new tablet copy session on tablet 3ab40fe08f714c99a5961cc2ad0915d3 from peer 88b77f6edc1c4a72b548bf47c05704f3 at {username='slave'} at 127.6.196.195:39073: session id = 88b77f6edc1c4a72b548bf47c05704f3-3ab40fe08f714c99a5961cc2ad0915d3
I20240502 14:14:53.760529 16421 raft_consensus.cc:2454] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:53.761021 13515 leader_election.cc:302] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:14:53.761327 15839 raft_consensus.cc:2781] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:53.761785 15839 raft_consensus.cc:686] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:53.762404 15839 consensus_queue.cc:237] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:53.765179 16442 tablet_copy_source_session.cc:205] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:53.767647 17009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ab40fe08f714c99a5961cc2ad0915d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.768626 16985 tablet_bootstrap.cc:492] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:53.774335 17005 tablet_copy_client.cc:308] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:14:53.776901 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet eab73d39b1f7455bba7aef5e515805c7 from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:56671)
I20240502 14:14:53.777055 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet eab73d39b1f7455bba7aef5e515805c7 from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:56671: session id = 8b47d4390f8343a493d6c00123785968-eab73d39b1f7455bba7aef5e515805c7
I20240502 14:14:53.780534 13339 tablet_copy_source_session.cc:205] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:53.781148 16987 tablet_bootstrap.cc:492] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:53.782025 17005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eab73d39b1f7455bba7aef5e515805c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.782848 13044 catalog_manager.cc:5484] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:53.791170 16958 tablet_bootstrap.cc:492] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:53.791762 16958 tablet_bootstrap.cc:492] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:14:53.792075 16958 ts_tablet_manager.cc:1367] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.183s user 0.054s sys 0.023s
I20240502 14:14:53.793409 16958 raft_consensus.cc:348] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:53.793673 16851 raft_consensus.cc:482] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:53.793756 16958 raft_consensus.cc:729] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:14:53.793889 16851 raft_consensus.cc:504] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:53.794032 16958 consensus_queue.cc:260] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:53.794803 16851 leader_election.cc:288] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:53.795746 16958 ts_tablet_manager.cc:1398] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 14:14:53.797668 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-ba519bb3fb534e7fb6106f402e27446d received from {username='slave'} at 127.6.196.200:56671
I20240502 14:14:53.797861 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 8b47d4390f8343a493d6c00123785968-ba519bb3fb534e7fb6106f402e27446d on tablet ba519bb3fb534e7fb6106f402e27446d with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:14:53.809388 13319 consensus_queue.cc:237] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:53.809767 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bf499e8c044a1886989a90c6bb69fa" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:14:53.809989 16021 raft_consensus.cc:2454] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:53.810541 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5bf499e8c044a1886989a90c6bb69fa" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:14:53.810756 16021 raft_consensus.cc:3037] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.814394 16021 raft_consensus.cc:2454] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:53.816026 17009 tablet_copy_client.cc:783] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:53.816375 17009 tablet_copy_client.cc:647] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:53.820020 17009 tablet_copy_client.cc:523] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:53.827113 16421 raft_consensus.cc:1203] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:53.830571 17005 tablet_copy_client.cc:783] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:53.831199 17005 tablet_copy_client.cc:647] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:53.831367 13445 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.831799 17009 tablet_bootstrap.cc:492] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:53.832001 16881 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:14:53.832955 13253 consensus_peers.cc:485] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5975e02205a43d5b93ed8ab51f8a0f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:53.835354 17005 tablet_copy_client.cc:523] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:53.835943 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dff473ffa2439580897011db7f8d38" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:53.836169 16155 raft_consensus.cc:2454] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:14:53.836776 16220 leader_election.cc:302] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:53.839553 13580 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:53.840147 16881 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:53.841552 17005 tablet_bootstrap.cc:492] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:53.842415 15881 consensus_queue.cc:1035] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.844997 16881 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:53.846849 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dff473ffa2439580897011db7f8d38" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:53.846983 16851 raft_consensus.cc:2781] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:53.847064 13187 raft_consensus.cc:2454] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:14:53.847095 16851 raft_consensus.cc:482] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:53.847209 16851 raft_consensus.cc:3037] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.847869 13446 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:53.850711 16851 raft_consensus.cc:504] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:53.851912 16851 leader_election.cc:288] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:53.852571 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dff473ffa2439580897011db7f8d38" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:53.852766 16155 raft_consensus.cc:3037] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.855103 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42dff473ffa2439580897011db7f8d38" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:53.855363 13187 raft_consensus.cc:3037] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.855355 16155 raft_consensus.cc:2454] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:14:53.857647 13044 catalog_manager.cc:5484] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:53.859395 16220 leader_election.cc:302] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:53.859927 16851 raft_consensus.cc:2781] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:53.860586 16851 raft_consensus.cc:686] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:14:53.862071 16851 consensus_queue.cc:237] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:53.862757 17009 tablet_bootstrap.cc:492] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:53.863168 17009 tablet_bootstrap.cc:492] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap complete.
I20240502 14:14:53.863219 13187 raft_consensus.cc:2454] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:14:53.863428 17009 ts_tablet_manager.cc:1367] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.004s
I20240502 14:14:53.864539 17009 raft_consensus.cc:348] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:14:53.864784 17009 raft_consensus.cc:729] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: LEARNER
I20240502 14:14:53.865005 17009 consensus_queue.cc:260] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:14:53.866209 13580 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:53.869988 17009 ts_tablet_manager.cc:1398] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:14:53.871634 16442 tablet_copy_service.cc:342] P 8b47d4390f8343a493d6c00123785968: Request end of tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-3ab40fe08f714c99a5961cc2ad0915d3 received from {username='slave'} at 127.6.196.195:39073
I20240502 14:14:53.871820 16442 tablet_copy_service.cc:433] P 8b47d4390f8343a493d6c00123785968: ending tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-3ab40fe08f714c99a5961cc2ad0915d3 on tablet 3ab40fe08f714c99a5961cc2ad0915d3 with peer 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:14:53.873415 13579 raft_consensus.cc:3037] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:53.877405 16977 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.908218 16804 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.940829 13036 catalog_manager.cc:5484] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:53.943087 17025 tablet_copy_client.cc:308] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:14:53.943958 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet 0ec7ae12c17247e9b8c03659435bd4e2 from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:56181)
I20240502 14:14:53.944134 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet 0ec7ae12c17247e9b8c03659435bd4e2 from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:56181: session id = 5297e816f782430d9a01220f0471a882-0ec7ae12c17247e9b8c03659435bd4e2
I20240502 14:14:53.947016 13470 tablet_copy_source_session.cc:205] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:53.948429 17025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec7ae12c17247e9b8c03659435bd4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:53.954546 17025 tablet_copy_client.cc:783] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:53.954866 17025 tablet_copy_client.cc:647] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:53.956310 17025 tablet_copy_client.cc:523] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:53.959502 17025 tablet_bootstrap.cc:492] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:53.969955 17005 tablet_bootstrap.cc:492] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:53.970392 17005 tablet_bootstrap.cc:492] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:14:53.970623 17005 ts_tablet_manager.cc:1367] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.129s user 0.016s sys 0.023s
I20240502 14:14:53.971331 17005 raft_consensus.cc:348] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:53.971534 17005 raft_consensus.cc:729] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:14:53.971724 17005 consensus_queue.cc:260] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:53.972292 17005 ts_tablet_manager.cc:1398] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:53.975904 16754 consensus_queue.cc:1035] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:53.976979 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-eab73d39b1f7455bba7aef5e515805c7 received from {username='slave'} at 127.6.196.200:56671
I20240502 14:14:53.977115 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 8b47d4390f8343a493d6c00123785968-eab73d39b1f7455bba7aef5e515805c7 on tablet eab73d39b1f7455bba7aef5e515805c7 with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:14:53.981595 13046 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 14:14:53.981782 13046 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240502 14:14:53.988420 13046 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet a187cc50f4cd45f1bcf841ea6f04a5e0
I20240502 14:14:53.989288 13561 tablet_service.cc:1514] Processing DeleteTablet for tablet a187cc50f4cd45f1bcf841ea6f04a5e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:53.989467 13046 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 1ca719ad9a7b4ccea1c9db20434b92eb
I20240502 14:14:53.989598 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet a187cc50f4cd45f1bcf841ea6f04a5e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:53.989898 16538 tablet_replica.cc:330] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:53.990108 16538 raft_consensus.cc:2229] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.990182 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet a187cc50f4cd45f1bcf841ea6f04a5e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:53.990345 16538 raft_consensus.cc:2258] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.990403 16526 tablet_replica.cc:330] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:53.990736 16526 raft_consensus.cc:2229] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:53.991032 16526 raft_consensus.cc:2258] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.991058 16538 ts_tablet_manager.cc:1875] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.991655 16526 ts_tablet_manager.cc:1875] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:53.995848 16979 tablet_replica.cc:330] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:53.996060 16979 raft_consensus.cc:2229] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:53.996209 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca719ad9a7b4ccea1c9db20434b92eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:53.996291 16979 raft_consensus.cc:2258] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:53.996934 16979 ts_tablet_manager.cc:1875] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.000365 16526 ts_tablet_manager.cc:1888] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.000506 16526 log.cc:1196] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/a187cc50f4cd45f1bcf841ea6f04a5e0
I20240502 14:14:54.000939 16526 ts_tablet_manager.cc:1909] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:54.001823 13561 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca719ad9a7b4ccea1c9db20434b92eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:54.002075 13046 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 953692b0134743c980d43fcaf30238e1
I20240502 14:14:54.002422 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet a187cc50f4cd45f1bcf841ea6f04a5e0 (table rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]) successfully deleted
I20240502 14:14:54.002553 16526 tablet_replica.cc:330] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:54.002781 16526 raft_consensus.cc:2229] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.002839 13561 tablet_service.cc:1514] Processing DeleteTablet for tablet 953692b0134743c980d43fcaf30238e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:54.003082 16526 raft_consensus.cc:2258] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.004094 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet 953692b0134743c980d43fcaf30238e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:54.004806 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 953692b0134743c980d43fcaf30238e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:54.005460 16526 ts_tablet_manager.cc:1875] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.005738 16979 ts_tablet_manager.cc:1888] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.006945 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ca719ad9a7b4ccea1c9db20434b92eb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:53 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:54.007241 16975 tablet_replica.cc:330] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:54.007493 16975 raft_consensus.cc:2229] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.007534 16538 ts_tablet_manager.cc:1888] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.007723 16975 raft_consensus.cc:2258] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.007931 16979 log.cc:1196] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/a187cc50f4cd45f1bcf841ea6f04a5e0
I20240502 14:14:54.007790 16538 log.cc:1196] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/a187cc50f4cd45f1bcf841ea6f04a5e0
I20240502 14:14:54.008620 16975 ts_tablet_manager.cc:1875] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.009145 16538 ts_tablet_manager.cc:1909] T a187cc50f4cd45f1bcf841ea6f04a5e0 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:54.009305 16979 ts_tablet_manager.cc:1909] T a187cc50f4cd45f1bcf841ea6f04a5e0 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:54.010677 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet a187cc50f4cd45f1bcf841ea6f04a5e0 (table rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]) successfully deleted
I20240502 14:14:54.011351 16526 ts_tablet_manager.cc:1888] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.011605 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet a187cc50f4cd45f1bcf841ea6f04a5e0 (table rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]) successfully deleted
I20240502 14:14:54.011904 16979 tablet_replica.cc:330] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:54.012118 16538 tablet_replica.cc:330] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:54.012341 16979 raft_consensus.cc:2229] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:54.012408 16526 log.cc:1196] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/1ca719ad9a7b4ccea1c9db20434b92eb
I20240502 14:14:54.012766 16979 raft_consensus.cc:2258] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.013427 16979 ts_tablet_manager.cc:1875] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.013679 16538 raft_consensus.cc:2229] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:54.013989 16526 ts_tablet_manager.cc:1909] T 1ca719ad9a7b4ccea1c9db20434b92eb P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:54.014029 16538 raft_consensus.cc:2258] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.014782 16538 ts_tablet_manager.cc:1875] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.014905 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 1ca719ad9a7b4ccea1c9db20434b92eb (table rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]) successfully deleted
I20240502 14:14:54.015209 16526 tablet_replica.cc:330] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:54.015374 16975 ts_tablet_manager.cc:1888] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.015451 16526 raft_consensus.cc:2229] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.015713 16526 raft_consensus.cc:2258] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.015801 16975 log.cc:1196] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/1ca719ad9a7b4ccea1c9db20434b92eb
I20240502 14:14:54.016379 16526 ts_tablet_manager.cc:1875] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.016515 16975 ts_tablet_manager.cc:1909] T 1ca719ad9a7b4ccea1c9db20434b92eb P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:54.018103 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 1ca719ad9a7b4ccea1c9db20434b92eb (table rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]) successfully deleted
I20240502 14:14:54.022311 16526 ts_tablet_manager.cc:1888] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.022442 16526 log.cc:1196] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/953692b0134743c980d43fcaf30238e1
I20240502 14:14:54.022832 16526 ts_tablet_manager.cc:1909] T 953692b0134743c980d43fcaf30238e1 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:54.023604 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 953692b0134743c980d43fcaf30238e1 (table rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]) successfully deleted
I20240502 14:14:54.044059 16538 ts_tablet_manager.cc:1888] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.050253 16979 ts_tablet_manager.cc:1888] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.044184 16538 log.cc:1196] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/953692b0134743c980d43fcaf30238e1
I20240502 14:14:54.050793 16538 ts_tablet_manager.cc:1909] T 953692b0134743c980d43fcaf30238e1 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:54.051000 16979 log.cc:1196] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/1ca719ad9a7b4ccea1c9db20434b92eb
I20240502 14:14:54.052309 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 953692b0134743c980d43fcaf30238e1 (table rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]) successfully deleted
I20240502 14:14:54.052605 16979 ts_tablet_manager.cc:1909] T 1ca719ad9a7b4ccea1c9db20434b92eb P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:54.053941 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 1ca719ad9a7b4ccea1c9db20434b92eb (table rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]) successfully deleted
I20240502 14:14:54.054301 16979 tablet_replica.cc:330] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:54.055737 16979 raft_consensus.cc:2229] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.056025 16979 raft_consensus.cc:2258] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.056752 16979 ts_tablet_manager.cc:1875] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.064568 17005 ts_tablet_manager.cc:902] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:54.067981 16979 ts_tablet_manager.cc:1888] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.068193 16979 log.cc:1196] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/953692b0134743c980d43fcaf30238e1
I20240502 14:14:54.068675 16979 ts_tablet_manager.cc:1909] T 953692b0134743c980d43fcaf30238e1 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:54.070088 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 953692b0134743c980d43fcaf30238e1 (table rebalance_test_table_14 [id=4f0d325b9ba44b7c98e5cc0c7452223a]) successfully deleted
I20240502 14:14:54.071249 13046 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:54.074048 13445 raft_consensus.cc:1203] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:54.086175 16155 raft_consensus.cc:1261] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:54.086655 13187 raft_consensus.cc:1261] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:54.086907 16839 consensus_queue.cc:1035] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:54.087431 16839 consensus_queue.cc:1035] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:54.100526 16985 tablet_bootstrap.cc:492] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:54.103338 13046 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet 42dff473ffa2439580897011db7f8d38 (table rebalancer_dynamic_table [id=89514fc00a3a4fa194eac77c3bcccedb]). Expected version 1 got 0
I20240502 14:14:54.108342 13036 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 42dff473ffa2439580897011db7f8d38 (table rebalancer_dynamic_table [id=89514fc00a3a4fa194eac77c3bcccedb]). Expected version 1 got 0
I20240502 14:14:54.109764 16985 tablet_bootstrap.cc:492] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: Bootstrap complete.
I20240502 14:14:54.110139 16985 ts_tablet_manager.cc:1367] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.342s user 0.052s sys 0.016s
I20240502 14:14:54.111029 16420 raft_consensus.cc:1203] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:54.112275 16985 raft_consensus.cc:348] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:54.112624 16985 raft_consensus.cc:729] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: LEARNER
I20240502 14:14:54.112954 16985 consensus_queue.cc:260] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:54.116608 17072 tablet.cc:1722] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:54.117141 17072 tablet.cc:1620] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:14:54.122179 16985 ts_tablet_manager.cc:1398] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 14:14:54.124737 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session 5297e816f782430d9a01220f0471a882-bbd12f1c9884497b82c941b7231645bb received from {username='slave'} at 127.6.196.199:53105
I20240502 14:14:54.124917 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session 5297e816f782430d9a01220f0471a882-bbd12f1c9884497b82c941b7231645bb on tablet bbd12f1c9884497b82c941b7231645bb with peer 5297e816f782430d9a01220f0471a882
I20240502 14:14:54.136776 13036 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=89514fc00a3a4fa194eac77c3bcccedb] alter complete (version 1)
I20240502 14:14:54.140748 16987 tablet_bootstrap.cc:492] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:54.141219 16987 tablet_bootstrap.cc:492] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:14:54.141528 16987 ts_tablet_manager.cc:1367] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.361s user 0.052s sys 0.011s
I20240502 14:14:54.142287 16987 raft_consensus.cc:348] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:54.142529 16987 raft_consensus.cc:729] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:14:54.142720 16987 consensus_queue.cc:260] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:54.143357 16987 ts_tablet_manager.cc:1398] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:54.143584 13046 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 42dff473ffa2439580897011db7f8d38 (table rebalancer_dynamic_table [id=89514fc00a3a4fa194eac77c3bcccedb]). Expected version 1 got 0
I20240502 14:14:54.144469 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-718223dd8fd3432cb174a53325dc784c received from {username='slave'} at 127.6.196.198:40385
I20240502 14:14:54.144681 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-718223dd8fd3432cb174a53325dc784c on tablet 718223dd8fd3432cb174a53325dc784c with peer c86f7440570f4796babfd44c3d7fd847
I20240502 14:14:54.148283 17025 tablet_bootstrap.cc:492] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:54.148744 17025 tablet_bootstrap.cc:492] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882: Bootstrap complete.
I20240502 14:14:54.149014 17025 ts_tablet_manager.cc:1367] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.190s user 0.048s sys 0.008s
I20240502 14:14:54.151899 17025 raft_consensus.cc:348] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:54.152192 17025 raft_consensus.cc:729] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: LEARNER
I20240502 14:14:54.152412 17025 consensus_queue.cc:260] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:54.152894 17025 ts_tablet_manager.cc:1398] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:54.109608 17073 tablet.cc:1722] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:54.154150 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session 5297e816f782430d9a01220f0471a882-0ec7ae12c17247e9b8c03659435bd4e2 received from {username='slave'} at 127.6.196.199:56181
I20240502 14:14:54.154266 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session 5297e816f782430d9a01220f0471a882-0ec7ae12c17247e9b8c03659435bd4e2 on tablet 0ec7ae12c17247e9b8c03659435bd4e2 with peer 5297e816f782430d9a01220f0471a882
I20240502 14:14:54.154551 17073 tablet.cc:1620] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:54.169555 17076 tablet.cc:1722] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:54.170830 17005 tablet_copy_client.cc:308] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:14:54.171775 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet e3be42ac139741d19dd872ab551bebae from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:49573)
I20240502 14:14:54.171950 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet e3be42ac139741d19dd872ab551bebae from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:49573: session id = 8b47d4390f8343a493d6c00123785968-e3be42ac139741d19dd872ab551bebae
I20240502 14:14:54.177160 13470 tablet_copy_source_session.cc:205] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:54.180043 13046 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet 42dff473ffa2439580897011db7f8d38 (table rebalancer_dynamic_table [id=89514fc00a3a4fa194eac77c3bcccedb]). Expected version 1 got 0
I20240502 14:14:54.186386 17005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3be42ac139741d19dd872ab551bebae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.209455 17076 tablet.cc:1620] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:14:54.214124 17005 tablet_copy_client.cc:783] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:54.217911 17005 tablet_copy_client.cc:647] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:54.220428 17005 tablet_copy_client.cc:523] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:54.224570 17005 tablet_bootstrap.cc:492] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:54.232829 16288 raft_consensus.cc:1203] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:54.239518 13036 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:54.254244 17076 tablet.cc:1722] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:54.254796 17076 tablet.cc:1620] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:14:54.255606 17072 tablet.cc:1722] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:54.256127 17072 tablet.cc:1620] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:14:54.260875 17073 tablet.cc:1722] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:54.261423 17073 tablet.cc:1620] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:54.275875 13036 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=89514fc00a3a4fa194eac77c3bcccedb] alter complete (version 2)
I20240502 14:14:54.280215 16204 consensus_queue.cc:1035] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:54.285578 17005 tablet_bootstrap.cc:492] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:54.286150 17005 tablet_bootstrap.cc:492] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:14:54.286501 17005 ts_tablet_manager.cc:1367] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.062s user 0.044s sys 0.009s
I20240502 14:14:54.287564 17005 raft_consensus.cc:348] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:54.287940 17005 raft_consensus.cc:729] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:14:54.288241 17005 consensus_queue.cc:260] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:54.289312 17005 ts_tablet_manager.cc:1398] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 14:14:54.290319 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-e3be42ac139741d19dd872ab551bebae received from {username='slave'} at 127.6.196.200:49573
I20240502 14:14:54.290525 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session 8b47d4390f8343a493d6c00123785968-e3be42ac139741d19dd872ab551bebae on tablet e3be42ac139741d19dd872ab551bebae with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:14:54.298862 15839 consensus_queue.cc:1035] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:54.359603 17074 raft_consensus.cc:1050] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:14:54.360714 17074 consensus_queue.cc:237] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:14:54.365442 16420 raft_consensus.cc:1261] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:54.365984 13445 raft_consensus.cc:1261] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:54.366551 16929 consensus_queue.cc:1035] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:54.367142 16929 consensus_queue.cc:1035] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:54.368700 13188 raft_consensus.cc:1261] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:54.369355 16881 consensus_queue.cc:1035] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:54.384359 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:54.391930 17119 ts_tablet_manager.cc:902] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:54.396564 17072 tablet.cc:1722] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:54.397853 16881 raft_consensus.cc:2932] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:54.402266 17073 tablet.cc:1722] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:54.408943 13446 raft_consensus.cc:2932] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:54.411789 13041 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:54.412853 13186 raft_consensus.cc:2932] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:54.415840 16420 raft_consensus.cc:2932] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:54.419174 17076 tablet.cc:1722] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:54.425225 13045 catalog_manager.cc:5484] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:54.432507 13041 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=89514fc00a3a4fa194eac77c3bcccedb] alter complete (version 3)
I20240502 14:14:54.435055 13319 consensus_queue.cc:237] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:14:54.438473 13186 raft_consensus.cc:1261] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:54.439203 16420 raft_consensus.cc:1261] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:54.441584 17074 consensus_queue.cc:1035] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:54.442056 17074 consensus_queue.cc:1035] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:54.446216 16881 raft_consensus.cc:2932] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:54.447232 16002 tablet_service.cc:1467] Processing CreateTablet for tablet e17e489fe5e04a758d9d9d0259159c64 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=296e981cee4f48689b6fcf85f9ab9562]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.447750 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17e489fe5e04a758d9d9d0259159c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.449844 16420 raft_consensus.cc:2932] T ba519bb3fb534e7fb6106f402e27446d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:54.450074 13561 tablet_service.cc:1467] Processing CreateTablet for tablet e17e489fe5e04a758d9d9d0259159c64 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=296e981cee4f48689b6fcf85f9ab9562]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.450085 13168 tablet_service.cc:1467] Processing CreateTablet for tablet e17e489fe5e04a758d9d9d0259159c64 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=296e981cee4f48689b6fcf85f9ab9562]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.450488 13561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17e489fe5e04a758d9d9d0259159c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.450548 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17e489fe5e04a758d9d9d0259159c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.454617 13167 tablet_service.cc:1467] Processing CreateTablet for tablet 9535cdbc4294429593012b088528c449 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=296e981cee4f48689b6fcf85f9ab9562]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.454989 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9535cdbc4294429593012b088528c449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.456775 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 9535cdbc4294429593012b088528c449 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=296e981cee4f48689b6fcf85f9ab9562]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.457108 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9535cdbc4294429593012b088528c449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.454643 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 8e69d81920d147a38e2542d22ba8352a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=296e981cee4f48689b6fcf85f9ab9562]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.457835 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 8e69d81920d147a38e2542d22ba8352a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=296e981cee4f48689b6fcf85f9ab9562]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.458076 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e69d81920d147a38e2542d22ba8352a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.458243 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e69d81920d147a38e2542d22ba8352a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.472257 13045 catalog_manager.cc:5484] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:54.476334 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet ba519bb3fb534e7fb6106f402e27446d with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:54.476753 16975 tablet_replica.cc:330] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:54.477108 16975 raft_consensus.cc:2229] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.477510 16975 raft_consensus.cc:2258] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.478463 16975 ts_tablet_manager.cc:1875] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:54.481460 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 8e69d81920d147a38e2542d22ba8352a (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=296e981cee4f48689b6fcf85f9ab9562]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.481859 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e69d81920d147a38e2542d22ba8352a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.481846 16003 tablet_service.cc:1467] Processing CreateTablet for tablet 9535cdbc4294429593012b088528c449 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=296e981cee4f48689b6fcf85f9ab9562]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:54.482223 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9535cdbc4294429593012b088528c449. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.481205 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba519bb3fb534e7fb6106f402e27446d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:54.489454 17132 tablet_bootstrap.cc:492] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:54.491498 17132 tablet_bootstrap.cc:654] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.492811 17134 tablet_bootstrap.cc:492] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:54.493340 16975 ts_tablet_manager.cc:1888] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:14:54.493463 16975 log.cc:1196] T ba519bb3fb534e7fb6106f402e27446d P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/ba519bb3fb534e7fb6106f402e27446d
I20240502 14:14:54.494163 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet ba519bb3fb534e7fb6106f402e27446d (table rebalance_test_table_15 [id=557f9d603bda4eb385d26221f3612c33]) successfully deleted
I20240502 14:14:54.495476 17134 tablet_bootstrap.cc:654] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.502179 17132 tablet_bootstrap.cc:492] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:54.502593 17119 tablet_copy_client.cc:308] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:14:54.502635 17132 ts_tablet_manager.cc:1367] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240502 14:14:54.503252 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet d5975e02205a43d5b93ed8ab51f8a0f8 from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:40779)
I20240502 14:14:54.503504 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet d5975e02205a43d5b93ed8ab51f8a0f8 from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:40779: session id = d1da7f11f0604f78934354a992bb0f55-d5975e02205a43d5b93ed8ab51f8a0f8
I20240502 14:14:54.503468 17132 raft_consensus.cc:348] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:54.503769 17132 raft_consensus.cc:374] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.503908 17132 raft_consensus.cc:729] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.504478 17132 consensus_queue.cc:260] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:54.504886 16288 raft_consensus.cc:1203] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:54.505190 17132 ts_tablet_manager.cc:1398] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:54.506153 13339 tablet_copy_source_session.cc:205] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:54.506861 17134 tablet_bootstrap.cc:492] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:54.507048 17134 ts_tablet_manager.cc:1367] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 14:14:54.508013 17134 raft_consensus.cc:348] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:54.508303 17134 raft_consensus.cc:374] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.508522 17134 raft_consensus.cc:729] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.508867 17134 consensus_queue.cc:260] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:54.509701 17134 ts_tablet_manager.cc:1398] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:54.510169 17134 tablet_bootstrap.cc:492] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:54.512130 13045 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 14:14:54.512310 13045 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240502 14:14:54.512881 17134 tablet_bootstrap.cc:654] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.514415 17119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5975e02205a43d5b93ed8ab51f8a0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.516112 13045 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet bd170e9091a84185ae8fa6714110d9f3
I20240502 14:14:54.516899 13045 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 4 replicas of tablet bbd12f1c9884497b82c941b7231645bb
I20240502 14:14:54.517122 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet bd170e9091a84185ae8fa6714110d9f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:54.517671 16975 tablet_replica.cc:330] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:54.518286 16975 raft_consensus.cc:2229] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:54.518579 13561 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd12f1c9884497b82c941b7231645bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:54.519281 16979 tablet_replica.cc:330] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:54.519588 16975 raft_consensus.cc:2258] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.519757 16979 raft_consensus.cc:2229] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:54.520280 16979 raft_consensus.cc:2258] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.520591 16975 ts_tablet_manager.cc:1875] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.520922 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd12f1c9884497b82c941b7231645bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:54.520999 16979 ts_tablet_manager.cc:1875] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.520931 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet bd170e9091a84185ae8fa6714110d9f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:54.525381 16538 tablet_replica.cc:330] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:54.525573 16538 raft_consensus.cc:2229] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.525909 16538 raft_consensus.cc:2258] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.526580 16538 ts_tablet_manager.cc:1875] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.528501 17119 tablet_copy_client.cc:783] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:54.528515 16265 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd12f1c9884497b82c941b7231645bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:14:54.528751 17119 tablet_copy_client.cc:647] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:54.530249 13045 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 624672f7de7843a595be3f75203f65e6
I20240502 14:14:54.530297 17119 tablet_copy_client.cc:523] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:54.530731 16267 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd12f1c9884497b82c941b7231645bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:14:54.533427 17119 tablet_bootstrap.cc:492] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:54.533797 16538 ts_tablet_manager.cc:1888] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:54.533905 16538 log.cc:1196] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/bbd12f1c9884497b82c941b7231645bb
I20240502 14:14:54.534329 16538 ts_tablet_manager.cc:1909] T bbd12f1c9884497b82c941b7231645bb P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:54.535193 16538 tablet_replica.cc:330] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:54.535261 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet bbd12f1c9884497b82c941b7231645bb (table rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]) successfully deleted
I20240502 14:14:54.535383 16538 raft_consensus.cc:2229] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.535595 16538 raft_consensus.cc:2258] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.536254 16538 ts_tablet_manager.cc:1875] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.537562 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet 624672f7de7843a595be3f75203f65e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:54.540159 13561 tablet_service.cc:1514] Processing DeleteTablet for tablet 624672f7de7843a595be3f75203f65e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:54.546245 17134 tablet_bootstrap.cc:492] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:54.546442 17134 ts_tablet_manager.cc:1367] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.005s
I20240502 14:14:54.547231 17134 raft_consensus.cc:348] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:54.547489 17134 raft_consensus.cc:374] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.547600 17134 raft_consensus.cc:729] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.547940 16979 ts_tablet_manager.cc:1888] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:54.547966 17134 consensus_queue.cc:260] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:54.548133 16979 log.cc:1196] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/bbd12f1c9884497b82c941b7231645bb
I20240502 14:14:54.548668 16979 ts_tablet_manager.cc:1909] T bbd12f1c9884497b82c941b7231645bb P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:54.548934 17134 ts_tablet_manager.cc:1398] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:54.549461 17134 tablet_bootstrap.cc:492] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:54.549894 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet bbd12f1c9884497b82c941b7231645bb (table rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]) successfully deleted
I20240502 14:14:54.551121 17134 tablet_bootstrap.cc:654] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.551365 16538 ts_tablet_manager.cc:1888] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.552340 16538 log.cc:1196] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/bd170e9091a84185ae8fa6714110d9f3
I20240502 14:14:54.552811 16538 ts_tablet_manager.cc:1909] T bd170e9091a84185ae8fa6714110d9f3 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:54.553170 16979 tablet_replica.cc:330] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:54.553467 16979 raft_consensus.cc:2229] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.553743 16979 raft_consensus.cc:2258] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.553848 16538 tablet_replica.cc:330] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:54.554018 16538 raft_consensus.cc:2229] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:54.554057 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet bd170e9091a84185ae8fa6714110d9f3 (table rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]) successfully deleted
I20240502 14:14:54.554329 16538 raft_consensus.cc:2258] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.554961 16538 ts_tablet_manager.cc:1875] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.555069 16979 ts_tablet_manager.cc:1875] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.556859 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 624672f7de7843a595be3f75203f65e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:54.557181 17148 tablet_bootstrap.cc:492] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:54.558966 17148 tablet_bootstrap.cc:654] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.560590 16975 ts_tablet_manager.cc:1888] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.562486 16267 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240502 14:14:54.563017 17134 tablet_bootstrap.cc:492] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:54.563213 17134 ts_tablet_manager.cc:1367] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 14:14:54.564340 17134 raft_consensus.cc:348] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:54.564574 17134 raft_consensus.cc:374] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.564692 17134 raft_consensus.cc:729] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.564963 17134 consensus_queue.cc:260] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:54.565644 17134 ts_tablet_manager.cc:1398] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:54.567404 17148 tablet_bootstrap.cc:492] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:54.567606 16975 log.cc:1196] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/bd170e9091a84185ae8fa6714110d9f3
I20240502 14:14:54.568646 16975 ts_tablet_manager.cc:1909] T bd170e9091a84185ae8fa6714110d9f3 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:54.570755 17148 ts_tablet_manager.cc:1367] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 14:14:54.571002 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet bd170e9091a84185ae8fa6714110d9f3 (table rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]) successfully deleted
I20240502 14:14:54.571740 17148 raft_consensus.cc:348] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:54.572094 17148 raft_consensus.cc:374] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.572222 17148 raft_consensus.cc:729] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.572717 17148 consensus_queue.cc:260] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:54.573029 13036 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:54.573307 13036 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:54.574167 16975 tablet_replica.cc:330] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:54.574842 17148 ts_tablet_manager.cc:1398] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:54.575567 16538 ts_tablet_manager.cc:1888] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.575774 16538 log.cc:1196] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/624672f7de7843a595be3f75203f65e6
I20240502 14:14:54.581121 13036 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 42dff473ffa2439580897011db7f8d38
I20240502 14:14:54.581480 16975 raft_consensus.cc:2229] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.582123 16538 ts_tablet_manager.cc:1909] T 624672f7de7843a595be3f75203f65e6 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:54.582373 16975 raft_consensus.cc:2258] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.583642 16267 tablet_service.cc:1514] Processing DeleteTablet for tablet 42dff473ffa2439580897011db7f8d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:14:54.583914 16975 ts_tablet_manager.cc:1875] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.584340 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 624672f7de7843a595be3f75203f65e6 (table rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]) successfully deleted
I20240502 14:14:54.585439 16979 ts_tablet_manager.cc:1888] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.585408 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet 42dff473ffa2439580897011db7f8d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:14:54.585567 16979 log.cc:1196] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/624672f7de7843a595be3f75203f65e6
I20240502 14:14:54.585987 16979 ts_tablet_manager.cc:1909] T 624672f7de7843a595be3f75203f65e6 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:54.586951 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 624672f7de7843a595be3f75203f65e6 (table rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]) successfully deleted
I20240502 14:14:54.604151 17146 tablet_bootstrap.cc:492] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:54.605894 17146 tablet_bootstrap.cc:654] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.607314 17152 tablet_replica.cc:330] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:14:54.607669 17152 raft_consensus.cc:2229] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.608000 17152 raft_consensus.cc:2258] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.609385 16975 ts_tablet_manager.cc:1888] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.609546 16975 log.cc:1196] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/624672f7de7843a595be3f75203f65e6
I20240502 14:14:54.610093 17152 ts_tablet_manager.cc:1875] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.610177 16975 ts_tablet_manager.cc:1909] T 624672f7de7843a595be3f75203f65e6 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:54.611245 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 624672f7de7843a595be3f75203f65e6 (table rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]) successfully deleted
I20240502 14:14:54.626415 17136 tablet_bootstrap.cc:492] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:54.629453 17136 tablet_bootstrap.cc:654] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.630069 17138 tablet_bootstrap.cc:492] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:54.632125 17138 tablet_bootstrap.cc:654] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.638868 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet bd170e9091a84185ae8fa6714110d9f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:54.638813 13186 raft_consensus.cc:2932] T ba519bb3fb534e7fb6106f402e27446d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:54.642154 13561 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:14:54.642603 13167 tablet_service.cc:1514] Processing DeleteTablet for tablet bbd12f1c9884497b82c941b7231645bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:54.642817 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet 42dff473ffa2439580897011db7f8d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:54 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:54.648592 16420 raft_consensus.cc:1203] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:54.661196 16399 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:14:54.667855 16155 raft_consensus.cc:1203] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:54.669519 16804 raft_consensus.cc:482] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:54.669659 16804 raft_consensus.cc:504] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:54.670293 16804 leader_election.cc:288] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:14:54.672870 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e69d81920d147a38e2542d22ba8352a" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:54.673080 13186 raft_consensus.cc:2454] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:14:54.673869 16088 leader_election.cc:302] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:54.674151 16804 raft_consensus.cc:2781] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:54.674243 16804 raft_consensus.cc:482] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:54.674319 16804 raft_consensus.cc:3037] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.675033 17146 tablet_bootstrap.cc:492] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:54.675166 17146 ts_tablet_manager.cc:1367] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.071s user 0.003s sys 0.000s
I20240502 14:14:54.633461 17143 tablet_replica.cc:330] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:14:54.670993 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e69d81920d147a38e2542d22ba8352a" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
W20240502 14:14:54.676420 16090 leader_election.cc:341] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:54.677352 16804 raft_consensus.cc:504] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:54.678884 16804 leader_election.cc:288] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:14:54.679147 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e69d81920d147a38e2542d22ba8352a" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:54.679441 13186 raft_consensus.cc:3037] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.682277 13186 raft_consensus.cc:2454] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:14:54.682618 17158 tablet_replica.cc:330] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:54.682878 17158 raft_consensus.cc:2229] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.682878 16088 leader_election.cc:302] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:54.682934 17057 raft_consensus.cc:482] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:54.683070 17057 raft_consensus.cc:504] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:54.683171 17158 raft_consensus.cc:2258] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.683686 16804 raft_consensus.cc:2781] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:54.683799 17057 leader_election.cc:288] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:54.683921 17158 ts_tablet_manager.cc:1875] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.685604 13036 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:54.686672 16804 raft_consensus.cc:686] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:14:54.687031 16804 consensus_queue.cc:237] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:54.677179 17143 raft_consensus.cc:2229] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:14:54.691833 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9535cdbc4294429593012b088528c449" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:54.678836 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e69d81920d147a38e2542d22ba8352a" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:14:54.692054 17143 raft_consensus.cc:2258] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Raft consensus is shut down!
W20240502 14:14:54.692772 16090 leader_election.cc:341] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:54.692826 17143 ts_tablet_manager.cc:1875] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.696293 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 240ea229031a427db69e64b03b32e9c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=de1e32fb39484256973df40013018336]), partition=
I20240502 14:14:54.696626 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240ea229031a427db69e64b03b32e9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.697618 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 240ea229031a427db69e64b03b32e9c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=de1e32fb39484256973df40013018336]), partition=
I20240502 14:14:54.698226 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240ea229031a427db69e64b03b32e9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.705866 17122 raft_consensus.cc:1050] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:14:54.706107 16003 tablet_service.cc:1467] Processing CreateTablet for tablet 240ea229031a427db69e64b03b32e9c9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=de1e32fb39484256973df40013018336]), partition=
I20240502 14:14:54.706408 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 240ea229031a427db69e64b03b32e9c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:54.706866 17122 consensus_queue.cc:237] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:14:54.720970 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9535cdbc4294429593012b088528c449" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:14:54.721124 17158 ts_tablet_manager.cc:1888] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:54.721271 17158 log.cc:1196] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/bd170e9091a84185ae8fa6714110d9f3
W20240502 14:14:54.721674 13122 leader_election.cc:341] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:54.721789 17158 ts_tablet_manager.cc:1909] T bd170e9091a84185ae8fa6714110d9f3 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:54.722617 17158 tablet_replica.cc:330] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:54.722826 17158 raft_consensus.cc:2229] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.723065 17158 raft_consensus.cc:2258] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.723141 16135 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:14:54.723737 17158 ts_tablet_manager.cc:1875] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.724040 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet bd170e9091a84185ae8fa6714110d9f3 (table rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]) successfully deleted
I20240502 14:14:54.725628 13168 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:14:54.729665 17158 ts_tablet_manager.cc:1888] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:54.729823 17158 log.cc:1196] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/bbd12f1c9884497b82c941b7231645bb
I20240502 14:14:54.730298 17158 ts_tablet_manager.cc:1909] T bbd12f1c9884497b82c941b7231645bb P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:54.731284 17158 tablet_replica.cc:330] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:54.731498 17158 raft_consensus.cc:2229] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:54.731693 17158 raft_consensus.cc:2258] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.735203 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet bbd12f1c9884497b82c941b7231645bb (table rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]) successfully deleted
I20240502 14:14:54.743548 17163 raft_consensus.cc:1050] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968: attempting to promote NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 to VOTER
I20240502 14:14:54.744953 17163 consensus_queue.cc:237] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:14:54.751495 17138 tablet_bootstrap.cc:492] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:54.751636 17138 ts_tablet_manager.cc:1367] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.122s user 0.003s sys 0.000s
I20240502 14:14:54.752470 17138 raft_consensus.cc:348] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:54.752671 17138 raft_consensus.cc:374] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.752768 17138 raft_consensus.cc:729] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.759065 17148 tablet_bootstrap.cc:492] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:54.755713 13044 catalog_manager.cc:5484] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:54.760692 17148 tablet_bootstrap.cc:654] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.766058 16420 raft_consensus.cc:1261] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:54.768900 17146 raft_consensus.cc:348] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:54.769097 17146 raft_consensus.cc:374] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.769187 17146 raft_consensus.cc:729] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.769448 17146 consensus_queue.cc:260] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:54.770031 17146 ts_tablet_manager.cc:1398] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.095s user 0.002s sys 0.000s
I20240502 14:14:54.770264 17146 tablet_bootstrap.cc:492] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:54.771936 13446 raft_consensus.cc:1261] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:54.771970 16929 consensus_queue.cc:1035] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:54.772951 13186 raft_consensus.cc:1261] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:54.773048 16288 raft_consensus.cc:1261] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:54.753063 17138 consensus_queue.cc:260] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:54.774477 17067 consensus_queue.cc:1035] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:54.775133 17067 consensus_queue.cc:1035] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:54.775650 17067 consensus_queue.cc:1035] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:54.799283 17152 ts_tablet_manager.cc:1888] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:54.799603 17079 raft_consensus.cc:1050] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER 5297e816f782430d9a01220f0471a882 to VOTER
I20240502 14:14:54.796041 16002 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:14:54.800873 17079 consensus_queue.cc:237] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:14:54.802907 17152 log.cc:1196] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/42dff473ffa2439580897011db7f8d38
I20240502 14:14:54.807979 17138 ts_tablet_manager.cc:1398] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20240502 14:14:54.808370 17138 tablet_bootstrap.cc:492] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:54.808650 17152 ts_tablet_manager.cc:1909] T 42dff473ffa2439580897011db7f8d38 P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:14:54.810063 17136 tablet_bootstrap.cc:492] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:54.810206 17136 ts_tablet_manager.cc:1367] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.184s user 0.001s sys 0.003s
I20240502 14:14:54.810082 17067 raft_consensus.cc:2932] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:54.810632 13186 raft_consensus.cc:2932] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:54.810956 17136 raft_consensus.cc:348] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:54.811210 17136 raft_consensus.cc:374] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.811305 17136 raft_consensus.cc:729] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.811511 17136 consensus_queue.cc:260] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:54.812129 17136 ts_tablet_manager.cc:1398] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:54.818557 17138 tablet_bootstrap.cc:654] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.822484 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 42dff473ffa2439580897011db7f8d38 (table rebalancer_dynamic_table_new_name [id=89514fc00a3a4fa194eac77c3bcccedb]) successfully deleted
I20240502 14:14:54.824802 17143 ts_tablet_manager.cc:1888] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:54.825003 17143 log.cc:1196] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/bbd12f1c9884497b82c941b7231645bb
I20240502 14:14:54.826141 13036 catalog_manager.cc:5484] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:54.824692 16288 raft_consensus.cc:2932] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:54.830163 17146 tablet_bootstrap.cc:654] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.837762 17143 ts_tablet_manager.cc:1909] T bbd12f1c9884497b82c941b7231645bb P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
W20240502 14:14:54.841782 13121 leader_election.cc:341] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:54.842041 13121 leader_election.cc:302] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55
I20240502 14:14:54.845173 17143 tablet_replica.cc:330] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:14:54.845674 13579 raft_consensus.cc:1261] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:54.845975 17143 raft_consensus.cc:2229] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:54.846611 17143 raft_consensus.cc:2258] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:54.847308 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet bbd12f1c9884497b82c941b7231645bb (table rebalance_test_table_16 [id=1ce45faa68624e3bb49f45d1ad39a0e5]) successfully deleted
W20240502 14:14:54.847652 13031 catalog_manager.cc:4797] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): delete failed for tablet bbd12f1c9884497b82c941b7231645bb because the tablet was not found. No further retry: Not found: Tablet not found: bbd12f1c9884497b82c941b7231645bb
I20240502 14:14:54.850176 17074 consensus_queue.cc:1035] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:54.854890 13427 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240502 14:14:54.858273 17158 ts_tablet_manager.cc:1875] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.859521 15839 raft_consensus.cc:482] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:54.859704 15839 raft_consensus.cc:504] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:54.860414 15839 leader_election.cc:288] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:54.862605 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17e489fe5e04a758d9d9d0259159c64" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:54.862833 13186 raft_consensus.cc:2454] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:54.863452 16851 raft_consensus.cc:482] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:54.863440 13514 leader_election.cc:302] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:14:54.863559 16851 raft_consensus.cc:504] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:54.863706 15839 raft_consensus.cc:2781] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:54.863780 15839 raft_consensus.cc:482] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:54.863871 15839 raft_consensus.cc:3037] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.864199 16851 leader_election.cc:288] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:54.864807 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e69d81920d147a38e2542d22ba8352a" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:54.865029 13186 raft_consensus.cc:2379] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5297e816f782430d9a01220f0471a882 in current term 1: Already voted for candidate c86f7440570f4796babfd44c3d7fd847 in this term.
I20240502 14:14:54.865949 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e69d81920d147a38e2542d22ba8352a" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:54.866564 16220 leader_election.cc:302] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5297e816f782430d9a01220f0471a882; no voters: 20a967bad5504b1e9c6fb720ecdd1625, c86f7440570f4796babfd44c3d7fd847
I20240502 14:14:54.866930 16851 raft_consensus.cc:3037] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.867621 15839 raft_consensus.cc:504] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:54.868405 15839 leader_election.cc:288] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:54.869021 13297 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240502 14:14:54.870779 16851 raft_consensus.cc:2726] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:54.876853 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17e489fe5e04a758d9d9d0259159c64" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
W20240502 14:14:54.877637 13516 leader_election.cc:341] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:54.877738 17057 raft_consensus.cc:2726] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:54.877843 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17e489fe5e04a758d9d9d0259159c64" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
W20240502 14:14:54.878302 13516 leader_election.cc:341] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:54.878321 16021 raft_consensus.cc:1261] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:54.880693 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17e489fe5e04a758d9d9d0259159c64" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:54.880993 13186 raft_consensus.cc:3037] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.882009 17138 tablet_bootstrap.cc:492] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:54.881913 16929 consensus_queue.cc:1035] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:54.882186 17138 ts_tablet_manager.cc:1367] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.000s
I20240502 14:14:54.885448 17138 raft_consensus.cc:348] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:54.886178 13186 raft_consensus.cc:2454] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:54.886399 17138 raft_consensus.cc:374] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.886615 17138 raft_consensus.cc:729] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.886646 13514 leader_election.cc:302] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters: d1da7f11f0604f78934354a992bb0f55
I20240502 14:14:54.886924 15839 raft_consensus.cc:2781] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:54.887048 15839 raft_consensus.cc:686] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:54.887079 17138 consensus_queue.cc:260] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
W20240502 14:14:54.887207 13121 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.196.200:43857, user_credentials={real_user=slave}} blocked reactor thread for 45456.2us
I20240502 14:14:54.887277 15839 consensus_queue.cc:237] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:54.887552 17138 ts_tablet_manager.cc:1398] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:14:54.888847 17138 tablet_bootstrap.cc:492] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:54.892058 17138 tablet_bootstrap.cc:654] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:54.903194 13580 raft_consensus.cc:1261] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:54.903901 17083 consensus_queue.cc:1035] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:54.906399 17119 tablet_bootstrap.cc:492] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:54.908720 17146 tablet_bootstrap.cc:492] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:54.908969 17146 ts_tablet_manager.cc:1367] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.139s user 0.010s sys 0.000s
I20240502 14:14:54.910888 13315 raft_consensus.cc:1261] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:54.909130 16929 raft_consensus.cc:2932] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:54.909817 17146 raft_consensus.cc:348] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:54.911196 17146 raft_consensus.cc:374] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.911334 17146 raft_consensus.cc:729] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.911999 17083 consensus_queue.cc:1035] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:54.912986 17158 ts_tablet_manager.cc:1888] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:54.913336 17146 consensus_queue.cc:260] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:54.914713 17146 ts_tablet_manager.cc:1398] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:14:54.916138 13579 raft_consensus.cc:2932] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:54.917455 17167 raft_consensus.cc:482] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:54.917591 17167 raft_consensus.cc:504] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:54.918962 17158 log.cc:1196] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/42dff473ffa2439580897011db7f8d38
I20240502 14:14:54.919461 17119 tablet_bootstrap.cc:492] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:14:54.920010 17158 ts_tablet_manager.cc:1909] T 42dff473ffa2439580897011db7f8d38 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:54.907972 17138 tablet_bootstrap.cc:492] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:54.921504 16420 raft_consensus.cc:2932] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:54.924638 17119 ts_tablet_manager.cc:1367] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.388s user 0.054s sys 0.014s
I20240502 14:14:54.924746 17138 ts_tablet_manager.cc:1367] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20240502 14:14:54.926847 17138 raft_consensus.cc:348] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:54.927127 17138 raft_consensus.cc:374] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:54.927783 17138 raft_consensus.cc:729] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:54.928079 17138 consensus_queue.cc:260] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:54.926648 13036 catalog_manager.cc:5484] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:54.928647 17138 ts_tablet_manager.cc:1398] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:54.930328 16288 raft_consensus.cc:1261] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:54.929101 17119 raft_consensus.cc:348] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:54.931063 17079 consensus_queue.cc:1035] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:54.932586 17119 raft_consensus.cc:729] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:14:54.932832 17119 consensus_queue.cc:260] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:54.933705 17119 ts_tablet_manager.cc:1398] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20240502 14:14:54.934105 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 42dff473ffa2439580897011db7f8d38 (table rebalancer_dynamic_table_new_name [id=89514fc00a3a4fa194eac77c3bcccedb]) successfully deleted
I20240502 14:14:54.934756 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-d5975e02205a43d5b93ed8ab51f8a0f8 received from {username='slave'} at 127.6.196.197:40779
I20240502 14:14:54.934866 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-d5975e02205a43d5b93ed8ab51f8a0f8 on tablet d5975e02205a43d5b93ed8ab51f8a0f8 with peer d1da7f11f0604f78934354a992bb0f55
I20240502 14:14:54.935451 17143 ts_tablet_manager.cc:1875] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:54.946637 16885 raft_consensus.cc:482] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:54.946803 16885 raft_consensus.cc:504] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:54.947846 16885 leader_election.cc:288] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:54.950718 13044 catalog_manager.cc:5484] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:54.962527 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9535cdbc4294429593012b088528c449" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:54.963227 16420 raft_consensus.cc:2454] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:14:54.963850 15956 leader_election.cc:302] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:54.964213 16885 raft_consensus.cc:2781] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:54.964360 16885 raft_consensus.cc:482] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:54.964465 16885 raft_consensus.cc:3037] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.969085 17143 ts_tablet_manager.cc:1888] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:54.969413 17143 log.cc:1196] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/42dff473ffa2439580897011db7f8d38
I20240502 14:14:54.970028 17143 ts_tablet_manager.cc:1909] T 42dff473ffa2439580897011db7f8d38 P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:14:54.973699 16023 raft_consensus.cc:1203] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:54.973898 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 42dff473ffa2439580897011db7f8d38 (table rebalancer_dynamic_table_new_name [id=89514fc00a3a4fa194eac77c3bcccedb]) successfully deleted
I20240502 14:14:54.981313 17079 raft_consensus.cc:2932] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
W20240502 14:14:54.986299 17181 log.cc:924] Time spent T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20240502 14:14:54.987023 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9535cdbc4294429593012b088528c449" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:54.987233 13186 raft_consensus.cc:2454] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:14:54.988086 16353 leader_election.cc:302] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:14:54.988540 17163 raft_consensus.cc:2781] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:54.988624 17163 raft_consensus.cc:482] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:54.988760 17163 raft_consensus.cc:3037] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.989601 16288 raft_consensus.cc:2932] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:54.990181 17167 leader_election.cc:288] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:54.991466 16021 raft_consensus.cc:2932] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:54.992635 16885 raft_consensus.cc:504] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:54.993548 17163 raft_consensus.cc:504] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:54.993747 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9535cdbc4294429593012b088528c449" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:14:54.993996 16420 raft_consensus.cc:2379] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1da7f11f0604f78934354a992bb0f55 in current term 1: Already voted for candidate 8b47d4390f8343a493d6c00123785968 in this term.
I20240502 14:14:54.994472 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9535cdbc4294429593012b088528c449" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:54.994617 13186 raft_consensus.cc:3037] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:54.995461 13319 consensus_queue.cc:237] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:14:54.995831 16885 leader_election.cc:288] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:54.996493 13315 raft_consensus.cc:2932] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:54.998028 16420 raft_consensus.cc:1261] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:54.998241 13579 raft_consensus.cc:1261] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:54.998317 17163 leader_election.cc:288] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:54.998945 16929 consensus_queue.cc:1035] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:54.999390 16929 consensus_queue.cc:1035] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:54.996764 13580 raft_consensus.cc:2932] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:55.002238 17074 raft_consensus.cc:2932] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:55.003938 13186 raft_consensus.cc:2454] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:14:55.005652 16353 leader_election.cc:302] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:14:55.006748 16421 raft_consensus.cc:2932] T eab73d39b1f7455bba7aef5e515805c7 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:55.010164 16885 raft_consensus.cc:482] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:55.010298 16885 raft_consensus.cc:504] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:55.010845 16885 leader_election.cc:288] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:55.011961 17167 raft_consensus.cc:2781] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:55.012133 17167 raft_consensus.cc:686] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:14:55.012357 17167 consensus_queue.cc:237] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:55.018065 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eab73d39b1f7455bba7aef5e515805c7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:55.019557 13579 raft_consensus.cc:2932] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:55.025283 17079 raft_consensus.cc:1050] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:14:55.025524 13046 catalog_manager.cc:5484] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:55.026082 17079 consensus_queue.cc:237] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:14:55.031175 16885 raft_consensus.cc:482] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:55.031366 16885 raft_consensus.cc:504] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:55.031988 16885 leader_election.cc:288] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:55.032701 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240ea229031a427db69e64b03b32e9c9" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
W20240502 14:14:55.033800 15956 leader_election.cc:341] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:14:55.036235 13031 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet eab73d39b1f7455bba7aef5e515805c7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20240502 14:14:55.036934 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240ea229031a427db69e64b03b32e9c9" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:55.037374 16421 raft_consensus.cc:2454] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:14:55.037721 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17e489fe5e04a758d9d9d0259159c64" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:14:55.038499 15956 leader_election.cc:302] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters: c86f7440570f4796babfd44c3d7fd847
I20240502 14:14:55.039043 16885 raft_consensus.cc:2781] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:55.039139 16885 raft_consensus.cc:482] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:55.039237 16885 raft_consensus.cc:3037] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.039187 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9535cdbc4294429593012b088528c449" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:55.039491 13186 raft_consensus.cc:2379] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1da7f11f0604f78934354a992bb0f55 in current term 1: Already voted for candidate 8b47d4390f8343a493d6c00123785968 in this term.
I20240502 14:14:55.040587 16420 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:55.041024 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9535cdbc4294429593012b088528c449" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:55.041303 13186 raft_consensus.cc:2379] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d1da7f11f0604f78934354a992bb0f55 in current term 1: Already voted for candidate 8b47d4390f8343a493d6c00123785968 in this term.
I20240502 14:14:55.041208 17082 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:55.042043 13044 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 14:14:55.042124 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17e489fe5e04a758d9d9d0259159c64" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:55.042203 13044 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240502 14:14:55.042392 13188 raft_consensus.cc:2379] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d1da7f11f0604f78934354a992bb0f55 in current term 1: Already voted for candidate 7eb4ecb2379b44c0954f2658c4a04350 in this term.
I20240502 14:14:55.043359 16885 raft_consensus.cc:504] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:55.043866 13186 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:55.044584 17082 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:55.043870 16885 leader_election.cc:288] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:55.045779 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240ea229031a427db69e64b03b32e9c9" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:55.046152 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240ea229031a427db69e64b03b32e9c9" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:14:55.046346 16420 raft_consensus.cc:3037] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.044612 15955 leader_election.cc:302] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968
I20240502 14:14:55.047930 16885 raft_consensus.cc:2726] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:14:55.048332 16420 raft_consensus.cc:2454] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:14:55.048440 15955 leader_election.cc:302] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:14:55.048936 17082 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:55.049877 13186 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:55.049165 15956 leader_election.cc:302] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
W20240502 14:14:55.051226 15956 leader_election.cc:341] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:55.051481 16885 raft_consensus.cc:2781] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:55.052186 16885 raft_consensus.cc:686] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:14:55.052757 16885 consensus_queue.cc:237] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:55.053289 16420 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:55.056308 13045 catalog_manager.cc:5484] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:55.057989 13044 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet b3b57323bd14448b9406ce53ada769e5
I20240502 14:14:55.058669 13044 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 9e18282dc36248139609c3ad062a0e6d
I20240502 14:14:55.059144 13044 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 4 replicas of tablet 718223dd8fd3432cb174a53325dc784c
I20240502 14:14:55.060055 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet 718223dd8fd3432cb174a53325dc784c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:14:55.060349 17152 tablet_replica.cc:330] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:14:55.060534 17152 raft_consensus.cc:2229] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Raft consensus shutting down.
I20240502 14:14:55.060693 17152 raft_consensus.cc:2258] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Raft consensus is shut down!
I20240502 14:14:55.060822 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e18282dc36248139609c3ad062a0e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:55.061657 17152 ts_tablet_manager.cc:1875] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:55.061774 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b57323bd14448b9406ce53ada769e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:55.061836 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e18282dc36248139609c3ad062a0e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:55.062803 13561 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b57323bd14448b9406ce53ada769e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:55.063091 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e18282dc36248139609c3ad062a0e6d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:55.063263 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 718223dd8fd3432cb174a53325dc784c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:55.063652 13044 catalog_manager.cc:5484] T eab73d39b1f7455bba7aef5e515805c7 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:55.063928 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet b3b57323bd14448b9406ce53ada769e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:55.064179 17158 tablet_replica.cc:330] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:55.064353 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 718223dd8fd3432cb174a53325dc784c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:55.064414 17158 raft_consensus.cc:2229] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:55.064635 17158 raft_consensus.cc:2258] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.065110 17189 raft_consensus.cc:3037] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.065289 17158 ts_tablet_manager.cc:1875] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:55.065377 13579 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:55.065358 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet 718223dd8fd3432cb174a53325dc784c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:55.065572 16538 tablet_replica.cc:330] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:55.065754 16538 raft_consensus.cc:2229] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:55.065929 16538 raft_consensus.cc:2258] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.066545 16538 ts_tablet_manager.cc:1875] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:55.067631 17189 raft_consensus.cc:2726] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:55.069399 16975 tablet_replica.cc:330] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:55.069578 16979 tablet_replica.cc:330] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:55.069608 17083 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:55.069783 16979 raft_consensus.cc:2229] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:55.070073 16979 raft_consensus.cc:2258] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.070691 16979 ts_tablet_manager.cc:1875] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:55.071346 16003 tablet_service.cc:1514] Processing DeleteTablet for tablet eab73d39b1f7455bba7aef5e515805c7 with delete_type TABLET_DATA_TOMBSTONED (TS d1da7f11f0604f78934354a992bb0f55 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41222
I20240502 14:14:55.072099 17158 ts_tablet_manager.cc:1888] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:55.072238 17158 log.cc:1196] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/b3b57323bd14448b9406ce53ada769e5
I20240502 14:14:55.072785 17158 ts_tablet_manager.cc:1909] T b3b57323bd14448b9406ce53ada769e5 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:55.073953 17158 tablet_replica.cc:330] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:55.074127 17158 raft_consensus.cc:2229] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:55.074375 17158 raft_consensus.cc:2258] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.075022 17158 ts_tablet_manager.cc:1875] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:55.076949 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet b3b57323bd14448b9406ce53ada769e5 (table rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]) successfully deleted
W20240502 14:14:55.078728 13252 consensus_peers.cc:485] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.081737 17158 ts_tablet_manager.cc:1888] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:55.081908 17158 log.cc:1196] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/718223dd8fd3432cb174a53325dc784c
I20240502 14:14:55.082439 17158 ts_tablet_manager.cc:1909] T 718223dd8fd3432cb174a53325dc784c P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:55.082547 13044 catalog_manager.cc:5484] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:55.083798 13045 catalog_manager.cc:5484] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:55.084161 16975 raft_consensus.cc:2229] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:55.084414 16975 raft_consensus.cc:2258] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.085170 16975 ts_tablet_manager.cc:1875] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:55.088721 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 718223dd8fd3432cb174a53325dc784c (table rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]) successfully deleted
I20240502 14:14:55.093776 16538 ts_tablet_manager.cc:1888] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:55.095489 17152 ts_tablet_manager.cc:1888] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:55.095690 16538 log.cc:1196] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/9e18282dc36248139609c3ad062a0e6d
I20240502 14:14:55.096041 16979 ts_tablet_manager.cc:1888] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:55.096629 16538 ts_tablet_manager.cc:1909] T 9e18282dc36248139609c3ad062a0e6d P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:55.096820 16979 log.cc:1196] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/b3b57323bd14448b9406ce53ada769e5
I20240502 14:14:55.097002 17152 log.cc:1196] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/718223dd8fd3432cb174a53325dc784c
I20240502 14:14:55.097865 17197 tablet_replica.cc:330] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:14:55.098148 17197 raft_consensus.cc:2229] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:55.098508 17197 raft_consensus.cc:2258] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.098996 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9535cdbc4294429593012b088528c449" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:14:55.099356 16021 raft_consensus.cc:2379] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8b47d4390f8343a493d6c00123785968 in current term 1: Already voted for candidate d1da7f11f0604f78934354a992bb0f55 in this term.
I20240502 14:14:55.099004 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9535cdbc4294429593012b088528c449" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:14:55.099686 16023 raft_consensus.cc:2379] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8b47d4390f8343a493d6c00123785968 in current term 1: Already voted for candidate d1da7f11f0604f78934354a992bb0f55 in this term.
I20240502 14:14:55.099752 17197 ts_tablet_manager.cc:1875] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:55.103037 16420 consensus_queue.cc:237] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:14:55.103565 16979 ts_tablet_manager.cc:1909] T b3b57323bd14448b9406ce53ada769e5 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:55.103776 17152 ts_tablet_manager.cc:1909] T 718223dd8fd3432cb174a53325dc784c P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:14:55.104887 16285 raft_consensus.cc:1261] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.105409 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet b3b57323bd14448b9406ce53ada769e5 (table rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]) successfully deleted
I20240502 14:14:55.106314 16538 tablet_replica.cc:330] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:55.106499 16538 raft_consensus.cc:2229] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:55.106492 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 718223dd8fd3432cb174a53325dc784c (table rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]) successfully deleted
I20240502 14:14:55.106891 16538 raft_consensus.cc:2258] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.106916 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 9e18282dc36248139609c3ad062a0e6d (table rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]) successfully deleted
I20240502 14:14:55.106887 17167 consensus_queue.cc:1035] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.107259 16979 tablet_replica.cc:330] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:55.107931 16979 raft_consensus.cc:2229] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:55.108278 16979 raft_consensus.cc:2258] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.108601 16538 ts_tablet_manager.cc:1875] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:55.108876 17159 raft_consensus.cc:2932] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:55.109464 16285 raft_consensus.cc:2932] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:55.109925 16979 ts_tablet_manager.cc:1875] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:55.111936 17148 tablet_bootstrap.cc:492] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:55.112097 17148 ts_tablet_manager.cc:1367] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.353s user 0.066s sys 0.134s
I20240502 14:14:55.112989 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3ab40fe08f714c99a5961cc2ad0915d3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:55.112905 17148 raft_consensus.cc:348] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:55.113138 17148 raft_consensus.cc:374] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.113237 17148 raft_consensus.cc:729] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.115865 13044 catalog_manager.cc:5484] T 3ab40fe08f714c99a5961cc2ad0915d3 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:55.118990 17148 consensus_queue.cc:260] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:55.120393 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ab40fe08f714c99a5961cc2ad0915d3 with delete_type TABLET_DATA_TOMBSTONED (TS 20a967bad5504b1e9c6fb720ecdd1625 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:55.120736 17158 tablet_replica.cc:330] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:55.120995 17158 raft_consensus.cc:2229] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:55.121397 17158 raft_consensus.cc:2258] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.122251 17158 ts_tablet_manager.cc:1875] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:55.120440 17148 ts_tablet_manager.cc:1398] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:14:55.128337 17158 ts_tablet_manager.cc:1888] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:14:55.128365 16538 ts_tablet_manager.cc:1888] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:55.128494 16538 log.cc:1196] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/b3b57323bd14448b9406ce53ada769e5
I20240502 14:14:55.128501 17158 log.cc:1196] T 3ab40fe08f714c99a5961cc2ad0915d3 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/3ab40fe08f714c99a5961cc2ad0915d3
I20240502 14:14:55.129067 16979 ts_tablet_manager.cc:1888] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:55.129199 16979 log.cc:1196] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/9e18282dc36248139609c3ad062a0e6d
I20240502 14:14:55.129333 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 3ab40fe08f714c99a5961cc2ad0915d3 (table rebalancer_extra_table_2 [id=7a3f9a47775b46e38274ca5c7b2dbe6d]) successfully deleted
I20240502 14:14:55.120039 13579 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:55.129684 16979 ts_tablet_manager.cc:1909] T 9e18282dc36248139609c3ad062a0e6d P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:55.130424 16979 tablet_replica.cc:330] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:55.130523 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 9e18282dc36248139609c3ad062a0e6d (table rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]) successfully deleted
I20240502 14:14:55.130674 16979 raft_consensus.cc:2229] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:55.130913 16979 raft_consensus.cc:2258] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.131475 16979 ts_tablet_manager.cc:1875] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:55.134961 16538 ts_tablet_manager.cc:1909] T b3b57323bd14448b9406ce53ada769e5 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:55.136238 16538 tablet_replica.cc:330] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:55.136471 16538 raft_consensus.cc:2229] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:55.137084 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet b3b57323bd14448b9406ce53ada769e5 (table rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]) successfully deleted
I20240502 14:14:55.136909 16538 raft_consensus.cc:2258] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.137913 16538 ts_tablet_manager.cc:1875] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:55.139598 13445 raft_consensus.cc:2932] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:55.142062 16975 ts_tablet_manager.cc:1888] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240502 14:14:55.144022 16538 ts_tablet_manager.cc:1888] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:55.146852 13446 consensus_queue.cc:237] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:14:55.148439 13447 raft_consensus.cc:1261] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.148994 13579 raft_consensus.cc:1261] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:55.149780 16288 raft_consensus.cc:1261] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:55.149849 17197 ts_tablet_manager.cc:1888] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:14:55.149892 17086 consensus_queue.cc:1035] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:55.149977 17197 log.cc:1196] T eab73d39b1f7455bba7aef5e515805c7 P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/eab73d39b1f7455bba7aef5e515805c7
I20240502 14:14:55.151049 17167 consensus_queue.cc:1035] T 3ab40fe08f714c99a5961cc2ad0915d3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.151245 16979 ts_tablet_manager.cc:1888] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:55.151364 16979 log.cc:1196] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/718223dd8fd3432cb174a53325dc784c
I20240502 14:14:55.151782 16979 ts_tablet_manager.cc:1909] T 718223dd8fd3432cb174a53325dc784c P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:55.152047 17083 consensus_queue.cc:1035] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:55.152741 16538 log.cc:1196] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/718223dd8fd3432cb174a53325dc784c
I20240502 14:14:55.152966 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 718223dd8fd3432cb174a53325dc784c (table rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]) successfully deleted
I20240502 14:14:55.153335 16538 ts_tablet_manager.cc:1909] T 718223dd8fd3432cb174a53325dc784c P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:55.154299 13447 raft_consensus.cc:2932] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:55.159251 16804 consensus_queue.cc:1035] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.160108 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 718223dd8fd3432cb174a53325dc784c (table rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]) successfully deleted
I20240502 14:14:55.160316 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet eab73d39b1f7455bba7aef5e515805c7 (table rebalancer_extra_table_0 [id=a6584ce357f343ffa6c92e7412b11020]) successfully deleted
I20240502 14:14:55.161454 17086 raft_consensus.cc:2932] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:55.165155 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0ec7ae12c17247e9b8c03659435bd4e2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:55.166182 13579 raft_consensus.cc:2932] T 0ec7ae12c17247e9b8c03659435bd4e2 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:55.170513 16975 log.cc:1196] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/9e18282dc36248139609c3ad062a0e6d
I20240502 14:14:55.170969 16975 ts_tablet_manager.cc:1909] T 9e18282dc36248139609c3ad062a0e6d P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:14:55.174665 16804 raft_consensus.cc:482] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:55.174849 16804 raft_consensus.cc:504] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:55.175498 16804 leader_election.cc:288] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:55.175892 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 9e18282dc36248139609c3ad062a0e6d (table rebalance_test_table_18 [id=9fb2dba02f0745699edbcaa1c4b1f4ed]) successfully deleted
I20240502 14:14:55.176317 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240ea229031a427db69e64b03b32e9c9" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:14:55.176455 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "240ea229031a427db69e64b03b32e9c9" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:55.176766 16420 raft_consensus.cc:2379] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c86f7440570f4796babfd44c3d7fd847 in current term 1: Already voted for candidate d1da7f11f0604f78934354a992bb0f55 in this term.
I20240502 14:14:55.177318 16089 leader_election.cc:302] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847; no voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55
I20240502 14:14:55.179097 16804 raft_consensus.cc:3037] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.182488 13044 catalog_manager.cc:5484] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:55.182837 16804 raft_consensus.cc:2726] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:14:55.187995 16804 consensus_queue.cc:1035] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.193425 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ec7ae12c17247e9b8c03659435bd4e2 with delete_type TABLET_DATA_TOMBSTONED (TS cd67493f6f0142f896010f2b8a39887d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:55.193887 16538 tablet_replica.cc:330] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:55.194202 16538 raft_consensus.cc:2229] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:55.194502 16538 raft_consensus.cc:2258] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.195366 16538 ts_tablet_manager.cc:1875] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:55.201514 16538 ts_tablet_manager.cc:1888] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:14:55.201683 16538 log.cc:1196] T 0ec7ae12c17247e9b8c03659435bd4e2 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/0ec7ae12c17247e9b8c03659435bd4e2
I20240502 14:14:55.202423 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 0ec7ae12c17247e9b8c03659435bd4e2 (table rebalance_test_table_13 [id=c0cb4710334148e4b976992ee52cd47c]) successfully deleted
I20240502 14:14:55.216938 16288 raft_consensus.cc:2932] T 0ec7ae12c17247e9b8c03659435bd4e2 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:55.239259 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:55.245893 16420 raft_consensus.cc:1261] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:55.246129 16155 raft_consensus.cc:1261] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:55.246734 17189 consensus_queue.cc:1035] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.247014 16885 consensus_queue.cc:1035] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.252915 13044 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 240ea229031a427db69e64b03b32e9c9 (table rebalancer_dynamic_table [id=de1e32fb39484256973df40013018336]). Expected version 1 got 0
I20240502 14:14:55.254520 17209 tablet.cc:1722] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:55.255219 17205 mvcc.cc:205] Tried to move back new op lower bound from 7023244473320259584 to 7023244472541913088. Current Snapshot: MvccSnapshot[applied={T|T < 7023244473320259584}]
I20240502 14:14:55.260229 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=de1e32fb39484256973df40013018336] alter complete (version 1)
I20240502 14:14:55.261103 17209 tablet.cc:1620] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:55.261744 17208 tablet.cc:1722] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:55.264073 17208 tablet.cc:1620] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:14:55.264441 13044 catalog_manager.cc:5576] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695) does not have the latest schema for tablet 240ea229031a427db69e64b03b32e9c9 (table rebalancer_dynamic_table [id=de1e32fb39484256973df40013018336]). Expected version 1 got 0
I20240502 14:14:55.274160 13044 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet 240ea229031a427db69e64b03b32e9c9 (table rebalancer_dynamic_table [id=de1e32fb39484256973df40013018336]). Expected version 1 got 0
I20240502 14:14:55.277245 17212 tablet.cc:1722] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:55.277904 17212 tablet.cc:1620] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:14:55.279637 13044 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet 240ea229031a427db69e64b03b32e9c9 (table rebalancer_dynamic_table [id=de1e32fb39484256973df40013018336]). Expected version 1 got 0
I20240502 14:14:55.336882 16155 consensus_queue.cc:237] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:14:55.339756 16023 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:14:55.339923 16090 consensus_peers.cc:485] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba8299c5f722440e92803100e7240723. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.340013 13579 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:55.340617 17164 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:55.340975 16804 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:55.343838 17164 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:14:55.344444 13579 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:14:55.350144 13036 catalog_manager.cc:5484] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:14:55.354151 16023 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:14:55.364292 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:55.368181 13579 consensus_queue.cc:237] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:55.372526 17208 tablet.cc:1722] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:55.372978 17208 tablet.cc:1620] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:14:55.373353 17209 tablet.cc:1722] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:55.373880 17209 tablet.cc:1620] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:55.381853 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=de1e32fb39484256973df40013018336] alter complete (version 2)
I20240502 14:14:55.383363 17212 tablet.cc:1722] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:55.383942 17212 tablet.cc:1620] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:14:55.385789 13447 raft_consensus.cc:1261] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:55.386242 16284 raft_consensus.cc:1261] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:55.386876 15839 consensus_queue.cc:1035] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:55.388041 15839 consensus_queue.cc:1035] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:55.402563 15839 consensus_queue.cc:1035] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.403501 17166 raft_consensus.cc:2932] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
W20240502 14:14:55.407486 13515 consensus_peers.cc:485] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16d6faae2add42c5bd4af6018a13b841. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.409832 13447 raft_consensus.cc:2932] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:55.412243 13044 catalog_manager.cc:5484] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:55.416771 16288 raft_consensus.cc:2932] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:55.423337 17166 consensus_queue.cc:1035] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.458346 13315 consensus_queue.cc:237] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:55.460283 13579 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.460439 13447 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:55.460462 13253 consensus_peers.cc:485] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a6b71f7e77b4084a8904b1045099472. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.460781 16929 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.461093 16881 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.463526 17122 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:55.464191 13447 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:55.464265 13579 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:55.468034 13036 catalog_manager.cc:5484] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:55.476804 17167 consensus_queue.cc:1035] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.481480 17167 consensus_queue.cc:1035] T 9535cdbc4294429593012b088528c449 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:55.486124 13447 consensus_queue.cc:237] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:55.488390 13188 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:55.488437 13383 consensus_peers.cc:485] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3e9160c46d64fa0a964cc39aec91dfd. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.488896 17082 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.488983 13315 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.489486 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:55.489919 17082 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.497794 17208 tablet.cc:1722] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:55.498332 17209 tablet.cc:1722] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:55.502352 17212 tablet.cc:1722] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:55.511345 17082 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:55.512676 13045 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=de1e32fb39484256973df40013018336] alter complete (version 3)
I20240502 14:14:55.514665 13319 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:55.516906 13188 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:55.517508 13045 catalog_manager.cc:5484] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:55.531577 13319 consensus_queue.cc:237] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:55.533901 13447 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.534022 13579 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.534484 16881 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.534940 17122 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.537140 16881 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:55.537809 13447 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
W20240502 14:14:55.540666 13253 consensus_peers.cc:485] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a95b9049a344b98be91b9961f3e5bee. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.543008 13044 catalog_manager.cc:5484] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:55.543437 13579 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:55.545320 16929 raft_consensus.cc:1050] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER d1da7f11f0604f78934354a992bb0f55 to VOTER
I20240502 14:14:55.546023 16929 consensus_queue.cc:237] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:14:55.548708 16023 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:55.548905 13447 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:55.548875 13580 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:55.549418 17074 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:55.549882 17074 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:55.551308 16929 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:55.558743 16929 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:55.562119 13580 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:55.562574 13446 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:55.563148 16023 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:55.567216 13036 catalog_manager.cc:5484] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:55.569120 13447 consensus_queue.cc:237] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:55.573983 13319 consensus_queue.cc:237] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:14:55.574195 13315 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.574359 13188 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.574687 17082 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.575137 17082 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.579161 16023 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:55.579778 16881 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:55.580651 13580 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:55.581297 17122 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:14:55.582079 13383 consensus_peers.cc:485] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7852539de07744caaf385620c5166861. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.584185 13579 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:55.583612 16929 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:55.589502 13036 catalog_manager.cc:5484] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:55.593773 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d5975e02205a43d5b93ed8ab51f8a0f8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:55.594238 17082 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:55.595693 16023 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:55.597378 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet d5975e02205a43d5b93ed8ab51f8a0f8 with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:55.598242 13186 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:55.599213 16975 tablet_replica.cc:330] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:55.599426 16975 raft_consensus.cc:2229] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:55.599681 16975 raft_consensus.cc:2258] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.600528 16975 ts_tablet_manager.cc:1875] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:55.604234 13044 catalog_manager.cc:5484] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:55.609074 16975 ts_tablet_manager.cc:1888] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:14:55.609195 16975 log.cc:1196] T d5975e02205a43d5b93ed8ab51f8a0f8 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/d5975e02205a43d5b93ed8ab51f8a0f8
I20240502 14:14:55.607805 13319 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:55.609968 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet d5975e02205a43d5b93ed8ab51f8a0f8 (table rebalance_test_table_13 [id=c0cb4710334148e4b976992ee52cd47c]) successfully deleted
I20240502 14:14:55.618817 13036 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:55.618999 13036 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:55.621651 13036 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 240ea229031a427db69e64b03b32e9c9
I20240502 14:14:55.622354 16003 tablet_service.cc:1514] Processing DeleteTablet for tablet 240ea229031a427db69e64b03b32e9c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:41222
I20240502 14:14:55.622507 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet 240ea229031a427db69e64b03b32e9c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:14:55.622668 17197 tablet_replica.cc:330] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:14:55.622730 16399 tablet_service.cc:1514] Processing DeleteTablet for tablet 240ea229031a427db69e64b03b32e9c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:55 UTC) from {username='slave'} at 127.0.0.1:53312
I20240502 14:14:55.622895 17197 raft_consensus.cc:2229] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:55.623427 17197 raft_consensus.cc:2258] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.624437 17197 ts_tablet_manager.cc:1875] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:55.635421 17197 ts_tablet_manager.cc:1888] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:55.635632 17197 log.cc:1196] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/240ea229031a427db69e64b03b32e9c9
I20240502 14:14:55.636271 17197 ts_tablet_manager.cc:1909] T 240ea229031a427db69e64b03b32e9c9 P d1da7f11f0604f78934354a992bb0f55: Deleting consensus metadata
I20240502 14:14:55.638276 17287 tablet_replica.cc:330] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:14:55.638528 17287 raft_consensus.cc:2229] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:55.638847 17287 raft_consensus.cc:2258] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.639737 17286 tablet_replica.cc:330] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:14:55.640041 17286 raft_consensus.cc:2229] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:55.640313 17286 raft_consensus.cc:2258] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.640270 13036 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:55.641738 17287 ts_tablet_manager.cc:1875] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:55.642382 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet 240ea229031a427db69e64b03b32e9c9 (table rebalancer_dynamic_table_new_name [id=de1e32fb39484256973df40013018336]) successfully deleted
I20240502 14:14:55.643914 17286 ts_tablet_manager.cc:1875] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:55.645401 13188 consensus_queue.cc:237] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:55.646992 17287 ts_tablet_manager.cc:1888] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:55.647147 17287 log.cc:1196] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/240ea229031a427db69e64b03b32e9c9
I20240502 14:14:55.647728 17287 ts_tablet_manager.cc:1909] T 240ea229031a427db69e64b03b32e9c9 P 8b47d4390f8343a493d6c00123785968: Deleting consensus metadata
I20240502 14:14:55.648897 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 240ea229031a427db69e64b03b32e9c9 (table rebalancer_dynamic_table_new_name [id=de1e32fb39484256973df40013018336]) successfully deleted
I20240502 14:14:55.650918 16265 tablet_service.cc:1467] Processing CreateTablet for tablet c871ce05c0804c2fbb3cb06049e6d56a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=89586fac1cc04eb6b38593443f23b4c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.651540 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c871ce05c0804c2fbb3cb06049e6d56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.651710 13168 tablet_service.cc:1467] Processing CreateTablet for tablet c871ce05c0804c2fbb3cb06049e6d56a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=89586fac1cc04eb6b38593443f23b4c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.651947 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 1d607f6e65194399844bc14457fe8f15 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=89586fac1cc04eb6b38593443f23b4c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.652108 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c871ce05c0804c2fbb3cb06049e6d56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.652370 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d607f6e65194399844bc14457fe8f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.654321 13447 consensus_queue.cc:237] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:14:55.655862 13560 tablet_service.cc:1467] Processing CreateTablet for tablet b648703b8fc742ecbe1dfd2047d4e68d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=89586fac1cc04eb6b38593443f23b4c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.656159 13579 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:55.656316 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b648703b8fc742ecbe1dfd2047d4e68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.656752 16420 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:55.656726 17086 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240502 14:14:55.658210 13122 consensus_peers.cc:485] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2905b7940fb45c49ef8b18943dae256. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.658252 17086 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:55.659322 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 1d607f6e65194399844bc14457fe8f15 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=89586fac1cc04eb6b38593443f23b4c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.659442 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 1d607f6e65194399844bc14457fe8f15 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=89586fac1cc04eb6b38593443f23b4c7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.659812 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d607f6e65194399844bc14457fe8f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.659871 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d607f6e65194399844bc14457fe8f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.662070 13319 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.663018 17086 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:55.665571 17286 ts_tablet_manager.cc:1888] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:55.665705 16003 tablet_service.cc:1467] Processing CreateTablet for tablet c871ce05c0804c2fbb3cb06049e6d56a (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=89586fac1cc04eb6b38593443f23b4c7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.666157 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c871ce05c0804c2fbb3cb06049e6d56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.666682 17286 log.cc:1196] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/240ea229031a427db69e64b03b32e9c9
I20240502 14:14:55.667093 17286 ts_tablet_manager.cc:1909] T 240ea229031a427db69e64b03b32e9c9 P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:14:55.670117 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 240ea229031a427db69e64b03b32e9c9 (table rebalancer_dynamic_table_new_name [id=de1e32fb39484256973df40013018336]) successfully deleted
I20240502 14:14:55.671195 13298 tablet_service.cc:1467] Processing CreateTablet for tablet b648703b8fc742ecbe1dfd2047d4e68d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=89586fac1cc04eb6b38593443f23b4c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.671306 17057 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.671612 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b648703b8fc742ecbe1dfd2047d4e68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.674476 16002 tablet_service.cc:1467] Processing CreateTablet for tablet b648703b8fc742ecbe1dfd2047d4e68d (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=89586fac1cc04eb6b38593443f23b4c7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:55.674991 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b648703b8fc742ecbe1dfd2047d4e68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.676788 16420 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:55.682802 13579 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:55.689677 17289 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:55.694164 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e3be42ac139741d19dd872ab551bebae with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:55.694196 13045 catalog_manager.cc:5484] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:55.695619 13315 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:55.699116 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet e3be42ac139741d19dd872ab551bebae with delete_type TABLET_DATA_TOMBSTONED (TS 20a967bad5504b1e9c6fb720ecdd1625 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:55.703811 13447 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.706025 17289 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.715660 17293 tablet_replica.cc:330] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:55.715427 13045 catalog_manager.cc:5484] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:55.716022 17293 raft_consensus.cc:2229] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:55.716432 17293 raft_consensus.cc:2258] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:55.724396 13045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:55.742022 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 798f5ac2da8d4172b154199bcd57d75d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=86de893985484748b7afe6e6f1de6b2c]), partition=
I20240502 14:14:55.742506 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798f5ac2da8d4172b154199bcd57d75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.746181 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 798f5ac2da8d4172b154199bcd57d75d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=86de893985484748b7afe6e6f1de6b2c]), partition=
I20240502 14:14:55.746548 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798f5ac2da8d4172b154199bcd57d75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.746568 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 798f5ac2da8d4172b154199bcd57d75d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=86de893985484748b7afe6e6f1de6b2c]), partition=
I20240502 14:14:55.746877 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 798f5ac2da8d4172b154199bcd57d75d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.761386 17293 ts_tablet_manager.cc:1875] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:55.771941 17300 tablet_bootstrap.cc:492] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:55.772620 17291 tablet_bootstrap.cc:492] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:55.774701 17299 tablet_bootstrap.cc:492] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:55.776823 17291 tablet_bootstrap.cc:654] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.778293 17299 tablet_bootstrap.cc:654] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.779668 17296 ts_tablet_manager.cc:902] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d: Initiating tablet copy from peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:55.780870 17297 tablet_bootstrap.cc:492] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:55.782960 17297 tablet_bootstrap.cc:654] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.783824 17293 ts_tablet_manager.cc:1888] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:14:55.783931 17293 log.cc:1196] T e3be42ac139741d19dd872ab551bebae P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/e3be42ac139741d19dd872ab551bebae
I20240502 14:14:55.784309 17298 tablet_bootstrap.cc:492] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:55.784528 17291 tablet_bootstrap.cc:492] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:55.784655 17291 ts_tablet_manager.cc:1367] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20240502 14:14:55.786156 17298 tablet_bootstrap.cc:654] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.785313 17291 raft_consensus.cc:348] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:55.786839 17291 raft_consensus.cc:374] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.786935 17291 raft_consensus.cc:729] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.787564 17291 consensus_queue.cc:260] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:55.788064 17291 ts_tablet_manager.cc:1398] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:55.787688 13447 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:55.789069 17291 tablet_bootstrap.cc:492] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:55.792194 17291 tablet_bootstrap.cc:654] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.792881 17300 tablet_bootstrap.cc:654] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.795843 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet e3be42ac139741d19dd872ab551bebae (table rebalance_test_table_5 [id=c97a45b4ee8c48a38ebd0709da5770f7]) successfully deleted
I20240502 14:14:55.798197 17299 tablet_bootstrap.cc:492] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:55.798415 17299 ts_tablet_manager.cc:1367] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20240502 14:14:55.799041 17298 tablet_bootstrap.cc:492] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:55.799203 17298 ts_tablet_manager.cc:1367] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240502 14:14:55.799831 17299 raft_consensus.cc:348] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:55.800082 17298 raft_consensus.cc:348] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:55.800313 17298 raft_consensus.cc:374] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.800407 17298 raft_consensus.cc:729] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.800652 17298 consensus_queue.cc:260] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:55.801004 17299 raft_consensus.cc:374] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.801286 17299 raft_consensus.cc:729] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.801756 17299 consensus_queue.cc:260] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:55.802052 17290 tablet_bootstrap.cc:492] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:55.802728 17297 tablet_bootstrap.cc:492] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:55.802843 17300 tablet_bootstrap.cc:492] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:55.802902 17297 ts_tablet_manager.cc:1367] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240502 14:14:55.803732 17300 ts_tablet_manager.cc:1367] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20240502 14:14:55.803694 17297 raft_consensus.cc:348] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:55.804497 17300 raft_consensus.cc:348] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:55.805110 17299 ts_tablet_manager.cc:1398] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:14:55.805253 17297 raft_consensus.cc:374] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.805388 17297 raft_consensus.cc:729] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.805680 17299 tablet_bootstrap.cc:492] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:55.805935 17297 consensus_queue.cc:260] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:55.806245 17290 tablet_bootstrap.cc:654] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.806771 17300 raft_consensus.cc:374] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.807010 17300 raft_consensus.cc:729] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.808566 17297 ts_tablet_manager.cc:1398] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:14:55.809095 17296 tablet_copy_client.cc:308] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.198:42041
I20240502 14:14:55.809433 17297 tablet_bootstrap.cc:492] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:55.809726 17300 consensus_queue.cc:260] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:55.810285 16175 tablet_copy_service.cc:140] P c86f7440570f4796babfd44c3d7fd847: Received BeginTabletCopySession request for tablet ba8299c5f722440e92803100e7240723 from peer cd67493f6f0142f896010f2b8a39887d ({username='slave'} at 127.6.196.194:59023)
I20240502 14:14:55.810578 16175 tablet_copy_service.cc:161] P c86f7440570f4796babfd44c3d7fd847: Beginning new tablet copy session on tablet ba8299c5f722440e92803100e7240723 from peer cd67493f6f0142f896010f2b8a39887d at {username='slave'} at 127.6.196.194:59023: session id = cd67493f6f0142f896010f2b8a39887d-ba8299c5f722440e92803100e7240723
I20240502 14:14:55.811275 17298 ts_tablet_manager.cc:1398] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:14:55.811374 17299 tablet_bootstrap.cc:654] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.812726 17300 ts_tablet_manager.cc:1398] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 14:14:55.813004 17300 tablet_bootstrap.cc:492] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:55.813565 17297 tablet_bootstrap.cc:654] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.815760 17300 tablet_bootstrap.cc:654] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.820658 16175 tablet_copy_source_session.cc:205] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:55.823755 17296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8299c5f722440e92803100e7240723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.830199 17300 tablet_bootstrap.cc:492] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:55.830492 17300 ts_tablet_manager.cc:1367] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.007s
I20240502 14:14:55.831629 17300 raft_consensus.cc:348] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:55.833061 17300 raft_consensus.cc:374] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.833318 17300 raft_consensus.cc:729] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.833894 17300 consensus_queue.cc:260] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:55.835633 17300 ts_tablet_manager.cc:1398] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20240502 14:14:55.836225 17300 tablet_bootstrap.cc:492] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:55.837698 16851 raft_consensus.cc:482] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:55.837944 16851 raft_consensus.cc:504] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:55.838645 16851 leader_election.cc:288] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:55.841441 17300 tablet_bootstrap.cc:654] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.841770 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c871ce05c0804c2fbb3cb06049e6d56a" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:55.841991 13188 raft_consensus.cc:2454] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:14:55.842489 16219 leader_election.cc:302] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 5297e816f782430d9a01220f0471a882; no voters:
I20240502 14:14:55.842870 15839 raft_consensus.cc:482] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:55.843014 15839 raft_consensus.cc:504] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:55.843564 15839 leader_election.cc:288] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:55.844256 17296 tablet_copy_client.cc:783] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:55.844533 17296 tablet_copy_client.cc:647] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:55.844914 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c871ce05c0804c2fbb3cb06049e6d56a" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
W20240502 14:14:55.845649 16221 leader_election.cc:341] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:55.846511 17296 tablet_copy_client.cc:523] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:55.846516 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b648703b8fc742ecbe1dfd2047d4e68d" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:55.847136 13315 raft_consensus.cc:2454] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:55.847806 13516 leader_election.cc:302] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:55.848134 17229 raft_consensus.cc:2781] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:55.848223 17229 raft_consensus.cc:482] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:55.848297 17229 raft_consensus.cc:3037] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.850989 17229 raft_consensus.cc:504] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:55.851560 17229 leader_election.cc:288] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:55.852013 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b648703b8fc742ecbe1dfd2047d4e68d" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:14:55.852272 16023 raft_consensus.cc:2454] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:55.852648 17297 tablet_bootstrap.cc:492] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:14:55.852690 16851 raft_consensus.cc:2781] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:55.852782 16851 raft_consensus.cc:482] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:55.852877 16851 raft_consensus.cc:3037] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.852972 17297 ts_tablet_manager.cc:1367] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.011s
I20240502 14:14:55.853993 17297 raft_consensus.cc:348] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:55.854305 17297 raft_consensus.cc:374] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.854426 17297 raft_consensus.cc:729] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.854722 17297 consensus_queue.cc:260] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:55.854966 17300 tablet_bootstrap.cc:492] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:55.855082 17290 tablet_bootstrap.cc:492] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:55.855422 17290 ts_tablet_manager.cc:1367] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.053s user 0.007s sys 0.007s
I20240502 14:14:55.855427 16851 raft_consensus.cc:504] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:55.855888 17299 tablet_bootstrap.cc:492] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:55.855111 17300 ts_tablet_manager.cc:1367] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.004s
I20240502 14:14:55.856098 17299 ts_tablet_manager.cc:1367] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.007s
I20240502 14:14:55.856474 17290 raft_consensus.cc:348] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:55.856709 17290 raft_consensus.cc:374] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.856894 17290 raft_consensus.cc:729] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.856915 17299 raft_consensus.cc:348] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:55.856988 17300 raft_consensus.cc:348] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:55.857213 17299 raft_consensus.cc:374] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.857342 17299 raft_consensus.cc:729] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.857192 17290 consensus_queue.cc:260] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:55.857213 17300 raft_consensus.cc:374] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.857633 17300 raft_consensus.cc:729] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.857786 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b648703b8fc742ecbe1dfd2047d4e68d" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:55.857844 17300 consensus_queue.cc:260] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:55.857975 13319 raft_consensus.cc:3037] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.857998 17290 ts_tablet_manager.cc:1398] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:55.857514 17296 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:55.858238 17290 tablet_bootstrap.cc:492] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:55.860863 13319 raft_consensus.cc:2454] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:55.861938 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b648703b8fc742ecbe1dfd2047d4e68d" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:14:55.862138 16021 raft_consensus.cc:3037] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.864743 17300 ts_tablet_manager.cc:1398] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:14:55.864773 16021 raft_consensus.cc:2454] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:55.864876 17290 tablet_bootstrap.cc:654] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:55.865886 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c871ce05c0804c2fbb3cb06049e6d56a" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:55.866115 13188 raft_consensus.cc:3037] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.868961 13188 raft_consensus.cc:2454] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:14:55.869429 16219 leader_election.cc:302] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 5297e816f782430d9a01220f0471a882; no voters:
I20240502 14:14:55.869968 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c871ce05c0804c2fbb3cb06049e6d56a" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:14:55.870203 17321 raft_consensus.cc:2781] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:55.870397 13516 leader_election.cc:302] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:55.871140 15839 raft_consensus.cc:2781] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:55.871280 15839 raft_consensus.cc:686] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:55.871562 15839 consensus_queue.cc:237] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:55.873430 17299 consensus_queue.cc:260] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:55.873831 16023 raft_consensus.cc:3037] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:55.876909 16023 raft_consensus.cc:2454] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:14:55.879323 17299 ts_tablet_manager.cc:1398] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240502 14:14:55.880730 17321 raft_consensus.cc:686] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:14:55.881124 17321 consensus_queue.cc:237] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:14:55.881796 17297 ts_tablet_manager.cc:1398] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240502 14:14:55.891922 16851 leader_election.cc:288] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:55.892452 13040 catalog_manager.cc:5484] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:55.892727 13045 catalog_manager.cc:5484] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:55.907092 17291 tablet_bootstrap.cc:492] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:55.908144 17291 ts_tablet_manager.cc:1367] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.119s user 0.014s sys 0.004s
I20240502 14:14:55.908650 17318 ts_tablet_manager.cc:902] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:55.912425 17296 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:55.913937 17291 raft_consensus.cc:348] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:55.915205 17296 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d: Bootstrap complete.
I20240502 14:14:55.915343 17291 raft_consensus.cc:374] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.915450 17291 raft_consensus.cc:729] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.915727 17291 consensus_queue.cc:260] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:55.915911 17296 ts_tablet_manager.cc:1367] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.059s user 0.022s sys 0.000s
I20240502 14:14:55.917312 17291 ts_tablet_manager.cc:1398] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 14:14:55.920200 17290 tablet_bootstrap.cc:492] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:55.920420 17290 ts_tablet_manager.cc:1367] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.007s
I20240502 14:14:55.921294 17290 raft_consensus.cc:348] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:55.921574 17290 raft_consensus.cc:374] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:55.921736 17290 raft_consensus.cc:729] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:55.922040 17290 consensus_queue.cc:260] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:55.922731 17290 ts_tablet_manager.cc:1398] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:14:55.923236 17296 raft_consensus.cc:348] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:14:55.923446 17296 raft_consensus.cc:729] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: LEARNER
I20240502 14:14:55.923681 17296 consensus_queue.cc:260] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:14:55.928192 17296 ts_tablet_manager.cc:1398] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240502 14:14:55.932343 16175 tablet_copy_service.cc:342] P c86f7440570f4796babfd44c3d7fd847: Request end of tablet copy session cd67493f6f0142f896010f2b8a39887d-ba8299c5f722440e92803100e7240723 received from {username='slave'} at 127.6.196.194:59023
I20240502 14:14:55.932498 16175 tablet_copy_service.cc:433] P c86f7440570f4796babfd44c3d7fd847: ending tablet copy session cd67493f6f0142f896010f2b8a39887d-ba8299c5f722440e92803100e7240723 on tablet ba8299c5f722440e92803100e7240723 with peer cd67493f6f0142f896010f2b8a39887d
I20240502 14:14:55.937417 13447 consensus_queue.cc:237] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:55.939337 17329 ts_tablet_manager.cc:902] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
W20240502 14:14:55.951181 13383 consensus_peers.cc:485] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: abd272455c064ba48726d933d53c83ed. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:55.951452 13188 raft_consensus.cc:1261] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.951952 17086 consensus_queue.cc:1035] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.960304 13319 raft_consensus.cc:1261] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:55.960987 17086 consensus_queue.cc:1035] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:55.973193 17086 raft_consensus.cc:2932] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:55.974568 17340 ts_tablet_manager.cc:902] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:55.976308 17337 ts_tablet_manager.cc:902] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:55.980497 17318 tablet_copy_client.cc:308] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:14:55.981339 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet 16d6faae2add42c5bd4af6018a13b841 from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:42449)
I20240502 14:14:55.981513 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet 16d6faae2add42c5bd4af6018a13b841 from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:42449: session id = c86f7440570f4796babfd44c3d7fd847-16d6faae2add42c5bd4af6018a13b841
I20240502 14:14:55.984596 17329 tablet_copy_client.cc:308] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:14:55.984946 13601 tablet_copy_source_session.cc:205] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:55.985168 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet c3e9160c46d64fa0a964cc39aec91dfd from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:49573)
I20240502 14:14:55.985312 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet c3e9160c46d64fa0a964cc39aec91dfd from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:49573: session id = 8b47d4390f8343a493d6c00123785968-c3e9160c46d64fa0a964cc39aec91dfd
I20240502 14:14:55.987919 13470 tablet_copy_source_session.cc:205] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:55.989260 17329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e9160c46d64fa0a964cc39aec91dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.990253 17318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d6faae2add42c5bd4af6018a13b841. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:55.993203 13040 catalog_manager.cc:5484] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:55.995265 13315 raft_consensus.cc:2932] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:55.997241 17318 tablet_copy_client.cc:783] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:55.997483 17318 tablet_copy_client.cc:647] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.006592 17329 tablet_copy_client.cc:783] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:56.006747 17318 tablet_copy_client.cc:523] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.006917 17329 tablet_copy_client.cc:647] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.008857 17329 tablet_copy_client.cc:523] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.009513 13188 raft_consensus.cc:2932] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:56.010996 17318 tablet_bootstrap.cc:492] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:56.014724 17329 tablet_bootstrap.cc:492] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:56.018049 16929 raft_consensus.cc:482] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:56.018298 16929 raft_consensus.cc:504] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:56.019245 16929 leader_election.cc:288] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:14:56.019706 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d607f6e65194399844bc14457fe8f15" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:56.020077 16155 raft_consensus.cc:2454] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:56.020247 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d607f6e65194399844bc14457fe8f15" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:14:56.020596 16284 raft_consensus.cc:2454] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:56.020865 13252 leader_election.cc:302] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:56.021353 16929 raft_consensus.cc:2781] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:56.021541 16929 raft_consensus.cc:482] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:56.021741 16929 raft_consensus.cc:3037] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.025209 16929 raft_consensus.cc:504] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:56.027125 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d607f6e65194399844bc14457fe8f15" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:56.027341 16155 raft_consensus.cc:3037] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.029487 16929 leader_election.cc:288] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:14:56.030113 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d607f6e65194399844bc14457fe8f15" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:14:56.030375 16284 raft_consensus.cc:3037] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.031036 16155 raft_consensus.cc:2454] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:56.031455 13252 leader_election.cc:302] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:56.032645 16929 raft_consensus.cc:2781] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:56.033576 16929 raft_consensus.cc:686] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:56.033878 16284 raft_consensus.cc:2454] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:56.033844 16929 consensus_queue.cc:237] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:56.036273 17318 tablet_bootstrap.cc:492] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:56.036842 17318 tablet_bootstrap.cc:492] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:14:56.037153 17318 ts_tablet_manager.cc:1367] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.026s user 0.011s sys 0.000s
I20240502 14:14:56.038153 17318 raft_consensus.cc:348] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:56.038447 17318 raft_consensus.cc:729] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:14:56.038683 17318 consensus_queue.cc:260] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:56.040380 17318 ts_tablet_manager.cc:1398] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:56.041126 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-16d6faae2add42c5bd4af6018a13b841 received from {username='slave'} at 127.6.196.198:42449
I20240502 14:14:56.041262 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-16d6faae2add42c5bd4af6018a13b841 on tablet 16d6faae2add42c5bd4af6018a13b841 with peer c86f7440570f4796babfd44c3d7fd847
I20240502 14:14:56.050858 13040 catalog_manager.cc:5484] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:56.060333 17337 tablet_copy_client.cc:308] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:14:56.060719 13188 consensus_queue.cc:237] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:56.061234 17340 tablet_copy_client.cc:308] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:14:56.061378 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 8a6b71f7e77b4084a8904b1045099472 from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:40779)
I20240502 14:14:56.061528 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 8a6b71f7e77b4084a8904b1045099472 from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:40779: session id = d1da7f11f0604f78934354a992bb0f55-8a6b71f7e77b4084a8904b1045099472
I20240502 14:14:56.063802 13339 tablet_copy_source_session.cc:205] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 14:14:56.065387 13122 consensus_peers.cc:485] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9c268e5a8e7462ab8e940775c73e591. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.065608 13579 raft_consensus.cc:1261] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:56.066082 17289 consensus_queue.cc:1035] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:56.068290 17057 raft_consensus.cc:2932] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:56.068966 17337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6b71f7e77b4084a8904b1045099472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.071740 13579 raft_consensus.cc:2932] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:56.081053 13040 catalog_manager.cc:5484] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:56.085109 17337 tablet_copy_client.cc:783] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:56.085261 16929 raft_consensus.cc:482] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:56.085431 16929 raft_consensus.cc:504] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:56.085569 17337 tablet_copy_client.cc:647] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.086297 16929 leader_election.cc:288] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:56.086994 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798f5ac2da8d4172b154199bcd57d75d" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:14:56.087234 13186 raft_consensus.cc:2454] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:56.087770 13251 leader_election.cc:302] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:56.088200 16929 raft_consensus.cc:2781] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:56.088325 16929 raft_consensus.cc:482] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:56.088420 16929 raft_consensus.cc:3037] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.089694 17337 tablet_copy_client.cc:523] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.091259 17329 tablet_bootstrap.cc:492] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:56.091274 16929 raft_consensus.cc:504] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:56.091886 17329 tablet_bootstrap.cc:492] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:14:56.092192 17329 ts_tablet_manager.cc:1367] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.078s user 0.046s sys 0.011s
I20240502 14:14:56.093031 17329 raft_consensus.cc:348] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:56.093290 17337 tablet_bootstrap.cc:492] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:56.093307 17329 raft_consensus.cc:729] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:14:56.093523 17329 consensus_queue.cc:260] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:56.094900 17329 ts_tablet_manager.cc:1398] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:56.095574 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-c3e9160c46d64fa0a964cc39aec91dfd received from {username='slave'} at 127.6.196.200:49573
I20240502 14:14:56.095690 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session 8b47d4390f8343a493d6c00123785968-c3e9160c46d64fa0a964cc39aec91dfd on tablet c3e9160c46d64fa0a964cc39aec91dfd with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:14:56.097503 13319 raft_consensus.cc:1203] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:56.098273 16929 leader_election.cc:288] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:56.099021 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798f5ac2da8d4172b154199bcd57d75d" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:56.099323 16155 raft_consensus.cc:2454] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:56.100152 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798f5ac2da8d4172b154199bcd57d75d" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:14:56.100394 13186 raft_consensus.cc:3037] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.104485 13186 raft_consensus.cc:2454] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:56.104928 13251 leader_election.cc:302] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:56.105209 16929 raft_consensus.cc:2781] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:56.105626 16929 raft_consensus.cc:686] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:56.105574 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "798f5ac2da8d4172b154199bcd57d75d" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:56.105825 16155 raft_consensus.cc:3037] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.105913 16929 consensus_queue.cc:237] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:56.112494 16155 raft_consensus.cc:2454] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:56.113565 17318 ts_tablet_manager.cc:902] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:56.121016 13040 catalog_manager.cc:5484] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:56.128703 16420 raft_consensus.cc:1261] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:56.129426 17289 consensus_queue.cc:1035] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:56.145079 16420 raft_consensus.cc:2932] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:56.152477 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 9a95b9049a344b98be91b9961f3e5bee from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:34141)
I20240502 14:14:56.152632 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 9a95b9049a344b98be91b9961f3e5bee from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:34141: session id = 5297e816f782430d9a01220f0471a882-9a95b9049a344b98be91b9961f3e5bee
I20240502 14:14:56.155071 13339 tablet_copy_source_session.cc:205] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:56.161686 17340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a95b9049a344b98be91b9961f3e5bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.165663 17318 tablet_copy_client.cc:308] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:14:56.169152 17340 tablet_copy_client.cc:783] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:56.169504 17340 tablet_copy_client.cc:647] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.171844 17340 tablet_copy_client.cc:523] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.175838 17340 tablet_bootstrap.cc:492] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:56.183282 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet 7852539de07744caaf385620c5166861 from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:52085)
I20240502 14:14:56.183429 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet 7852539de07744caaf385620c5166861 from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:52085: session id = c86f7440570f4796babfd44c3d7fd847-7852539de07744caaf385620c5166861
I20240502 14:14:56.186090 13470 tablet_copy_source_session.cc:205] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:56.187898 17318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7852539de07744caaf385620c5166861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.193943 17318 tablet_copy_client.cc:783] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:56.194223 17318 tablet_copy_client.cc:647] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.195663 17318 tablet_copy_client.cc:523] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.200181 17318 tablet_bootstrap.cc:492] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:56.213068 13579 consensus_queue.cc:237] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:56.224886 17337 tablet_bootstrap.cc:492] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:56.225419 17337 tablet_bootstrap.cc:492] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:14:56.225742 17337 ts_tablet_manager.cc:1367] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.133s user 0.049s sys 0.006s
I20240502 14:14:56.226612 17337 raft_consensus.cc:348] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:56.226928 17337 raft_consensus.cc:729] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:14:56.227175 17337 consensus_queue.cc:260] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:56.227840 17337 ts_tablet_manager.cc:1398] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:56.233922 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-8a6b71f7e77b4084a8904b1045099472 received from {username='slave'} at 127.6.196.197:40779
I20240502 14:14:56.234052 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-8a6b71f7e77b4084a8904b1045099472 on tablet 8a6b71f7e77b4084a8904b1045099472 with peer d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:56.235193 13515 consensus_peers.cc:485] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 652e49046212458280095c016a342f47. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:56.235294 17380 ts_tablet_manager.cc:902] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:56.235913 13319 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:56.237735 13447 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:56.238587 17229 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.245458 15839 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.248672 13045 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:56.282091 17340 tablet_bootstrap.cc:492] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:56.282562 17340 tablet_bootstrap.cc:492] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882: Bootstrap complete.
I20240502 14:14:56.282917 17340 ts_tablet_manager.cc:1367] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.107s user 0.070s sys 0.021s
I20240502 14:14:56.285070 17340 raft_consensus.cc:348] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:56.285401 17340 raft_consensus.cc:729] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: LEARNER
I20240502 14:14:56.285619 17340 consensus_queue.cc:260] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:56.286223 17340 ts_tablet_manager.cc:1398] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:14:56.286967 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 5297e816f782430d9a01220f0471a882-9a95b9049a344b98be91b9961f3e5bee received from {username='slave'} at 127.6.196.199:34141
I20240502 14:14:56.287070 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 5297e816f782430d9a01220f0471a882-9a95b9049a344b98be91b9961f3e5bee on tablet 9a95b9049a344b98be91b9961f3e5bee with peer 5297e816f782430d9a01220f0471a882
I20240502 14:14:56.298556 16155 raft_consensus.cc:1261] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:56.298606 13186 raft_consensus.cc:1261] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:56.299564 16881 consensus_queue.cc:1035] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.299962 16881 consensus_queue.cc:1035] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.302182 15839 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:56.305867 16154 raft_consensus.cc:1203] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:56.309176 13319 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:56.321925 16420 raft_consensus.cc:1203] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:56.325121 13044 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 798f5ac2da8d4172b154199bcd57d75d (table rebalancer_dynamic_table [id=86de893985484748b7afe6e6f1de6b2c]). Expected version 1 got 0
I20240502 14:14:56.331557 13447 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:56.336850 17166 consensus_queue.cc:1035] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20240502 14:14:56.338618 17380 tablet_copy_client.cc:308] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:36687
I20240502 14:14:56.339552 13208 tablet_copy_service.cc:140] P 20a967bad5504b1e9c6fb720ecdd1625: Received BeginTabletCopySession request for tablet b2905b7940fb45c49ef8b18943dae256 from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:57233)
I20240502 14:14:56.339743 13208 tablet_copy_service.cc:161] P 20a967bad5504b1e9c6fb720ecdd1625: Beginning new tablet copy session on tablet b2905b7940fb45c49ef8b18943dae256 from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:57233: session id = d1da7f11f0604f78934354a992bb0f55-b2905b7940fb45c49ef8b18943dae256
I20240502 14:14:56.343292 13208 tablet_copy_source_session.cc:205] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:56.343991 13040 catalog_manager.cc:5484] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:56.345067 17380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2905b7940fb45c49ef8b18943dae256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.347481 17318 tablet_bootstrap.cc:492] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:56.352612 17380 tablet_copy_client.cc:783] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:56.352901 17380 tablet_copy_client.cc:647] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.355597 17380 tablet_copy_client.cc:523] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.358868 17380 tablet_bootstrap.cc:492] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:56.363243 17229 consensus_queue.cc:1035] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.369043 13044 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet 798f5ac2da8d4172b154199bcd57d75d (table rebalancer_dynamic_table [id=86de893985484748b7afe6e6f1de6b2c]). Expected version 1 got 0
I20240502 14:14:56.391345 17318 tablet_bootstrap.cc:492] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:14:56.391703 17318 ts_tablet_manager.cc:1367] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.192s user 0.062s sys 0.008s
I20240502 14:14:56.392634 17318 raft_consensus.cc:348] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:56.392957 17318 raft_consensus.cc:729] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:14:56.393189 17318 consensus_queue.cc:260] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:56.400698 17397 tablet.cc:1722] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:56.402848 13045 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 798f5ac2da8d4172b154199bcd57d75d (table rebalancer_dynamic_table [id=86de893985484748b7afe6e6f1de6b2c]). Expected version 1 got 0
I20240502 14:14:56.407368 17318 ts_tablet_manager.cc:1398] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 14:14:56.409536 17400 tablet.cc:1722] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:56.410005 17400 tablet.cc:1620] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:14:56.412765 17397 tablet.cc:1620] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:56.412860 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-7852539de07744caaf385620c5166861 received from {username='slave'} at 127.6.196.198:52085
I20240502 14:14:56.413012 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-7852539de07744caaf385620c5166861 on tablet 7852539de07744caaf385620c5166861 with peer c86f7440570f4796babfd44c3d7fd847
I20240502 14:14:56.413679 17396 tablet.cc:1722] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:56.415030 17396 tablet.cc:1620] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: MemRowSet was empty: no flush needed.
W20240502 14:14:56.418260 13516 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.197:43695, user_credentials={real_user=slave}} blocked reactor thread for 55517.2us
I20240502 14:14:56.423858 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=86de893985484748b7afe6e6f1de6b2c] alter complete (version 1)
I20240502 14:14:56.444377 13045 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet 798f5ac2da8d4172b154199bcd57d75d (table rebalancer_dynamic_table [id=86de893985484748b7afe6e6f1de6b2c]). Expected version 1 got 0
I20240502 14:14:56.447599 17329 ts_tablet_manager.cc:902] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:56.470780 17380 tablet_bootstrap.cc:492] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:56.471581 17380 tablet_bootstrap.cc:492] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:14:56.471855 17380 ts_tablet_manager.cc:1367] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.113s user 0.047s sys 0.027s
I20240502 14:14:56.472630 17380 raft_consensus.cc:348] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:56.472852 17380 raft_consensus.cc:729] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:14:56.473011 17380 consensus_queue.cc:260] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:56.473665 17380 ts_tablet_manager.cc:1398] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:56.474349 13208 tablet_copy_service.cc:342] P 20a967bad5504b1e9c6fb720ecdd1625: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-b2905b7940fb45c49ef8b18943dae256 received from {username='slave'} at 127.6.196.197:57233
I20240502 14:14:56.474503 13208 tablet_copy_service.cc:433] P 20a967bad5504b1e9c6fb720ecdd1625: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-b2905b7940fb45c49ef8b18943dae256 on tablet b2905b7940fb45c49ef8b18943dae256 with peer d1da7f11f0604f78934354a992bb0f55
I20240502 14:14:56.476506 16851 consensus_queue.cc:1035] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.485507 17329 tablet_copy_client.cc:308] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:14:56.486208 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet abd272455c064ba48726d933d53c83ed from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:49573)
I20240502 14:14:56.486415 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet abd272455c064ba48726d933d53c83ed from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:49573: session id = 8b47d4390f8343a493d6c00123785968-abd272455c064ba48726d933d53c83ed
I20240502 14:14:56.490204 13470 tablet_copy_source_session.cc:205] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:56.491796 17329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd272455c064ba48726d933d53c83ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.498173 17329 tablet_copy_client.cc:783] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:56.498399 17329 tablet_copy_client.cc:647] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.501999 17329 tablet_copy_client.cc:523] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.506536 17329 tablet_bootstrap.cc:492] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:56.507476 17419 consensus_queue.cc:1035] T c871ce05c0804c2fbb3cb06049e6d56a P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.514734 17164 raft_consensus.cc:1050] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847: attempting to promote NON_VOTER cd67493f6f0142f896010f2b8a39887d to VOTER
I20240502 14:14:56.515718 17164 consensus_queue.cc:237] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:14:56.519959 13580 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:56.523029 13315 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:56.523665 17128 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:56.524207 16804 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:56.529958 16023 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:56.530535 16804 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:56.533721 17438 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:14:56.535558 13045 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:56.536935 13315 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:14:56.537979 13580 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:14:56.541918 16023 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:14:56.544595 16155 raft_consensus.cc:1203] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:56.545192 13044 catalog_manager.cc:5484] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:14:56.547941 16284 raft_consensus.cc:1203] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:56.550901 17396 tablet.cc:1722] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:56.551316 17396 tablet.cc:1620] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: MemRowSet was empty: no flush needed.
I20240502 14:14:56.552561 16021 raft_consensus.cc:1203] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:56.556444 17397 tablet.cc:1722] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:56.556634 17400 tablet.cc:1722] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:56.556913 17397 tablet.cc:1620] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:56.557049 17400 tablet.cc:1620] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:14:56.559738 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=86de893985484748b7afe6e6f1de6b2c] alter complete (version 2)
I20240502 14:14:56.564522 16155 consensus_queue.cc:237] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:14:56.567629 13315 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:56.568487 16021 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:56.568984 16804 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.569352 17164 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:56.572010 17164 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:14:56.573772 16021 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:14:56.582708 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba8299c5f722440e92803100e7240723 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:56.585026 13040 catalog_manager.cc:5484] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 3 to 4, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:56.588455 13315 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:14:56.589944 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet ba8299c5f722440e92803100e7240723 with delete_type TABLET_DATA_TOMBSTONED (TS 7eb4ecb2379b44c0954f2658c4a04350 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:56.590873 13044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:56.593246 17452 tablet_replica.cc:330] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:56.593484 17452 raft_consensus.cc:2229] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:56.593814 17329 tablet_bootstrap.cc:492] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:56.593864 17452 raft_consensus.cc:2258] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:56.594379 17329 tablet_bootstrap.cc:492] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:14:56.594628 17329 ts_tablet_manager.cc:1367] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.088s user 0.057s sys 0.001s
I20240502 14:14:56.594996 17452 ts_tablet_manager.cc:1875] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:56.595898 17329 raft_consensus.cc:348] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:56.596158 17329 raft_consensus.cc:729] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:14:56.596431 17329 consensus_queue.cc:260] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:56.596937 17329 ts_tablet_manager.cc:1398] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:56.597757 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-abd272455c064ba48726d933d53c83ed received from {username='slave'} at 127.6.196.200:49573
I20240502 14:14:56.597899 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session 8b47d4390f8343a493d6c00123785968-abd272455c064ba48726d933d53c83ed on tablet abd272455c064ba48726d933d53c83ed with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:14:56.600222 16021 raft_consensus.cc:1203] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:56.600682 17340 ts_tablet_manager.cc:902] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:14:56.605621 16929 consensus_queue.cc:1035] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.609174 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 8a98395ed74c4c2eb265a141aec14bd3 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ef46a85b1ca942a9ae1f3a3aad6af281]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.609644 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a98395ed74c4c2eb265a141aec14bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.610735 17452 ts_tablet_manager.cc:1888] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:14:56.610985 17452 log.cc:1196] T ba8299c5f722440e92803100e7240723 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/ba8299c5f722440e92803100e7240723
I20240502 14:14:56.612036 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet ba8299c5f722440e92803100e7240723 (table rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]) successfully deleted
I20240502 14:14:56.613520 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 819e6e83e6064c539cfc73a28c43db5b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ef46a85b1ca942a9ae1f3a3aad6af281]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.613888 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 819e6e83e6064c539cfc73a28c43db5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.615020 16003 tablet_service.cc:1467] Processing CreateTablet for tablet eb27b3bb29754d7399b0314ada6b1ba7 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ef46a85b1ca942a9ae1f3a3aad6af281]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.615381 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb27b3bb29754d7399b0314ada6b1ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.617887 16265 tablet_service.cc:1467] Processing CreateTablet for tablet 819e6e83e6064c539cfc73a28c43db5b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ef46a85b1ca942a9ae1f3a3aad6af281]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.618088 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 819e6e83e6064c539cfc73a28c43db5b (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ef46a85b1ca942a9ae1f3a3aad6af281]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.618337 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 819e6e83e6064c539cfc73a28c43db5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.618422 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 819e6e83e6064c539cfc73a28c43db5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.625074 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 8a98395ed74c4c2eb265a141aec14bd3 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ef46a85b1ca942a9ae1f3a3aad6af281]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.625478 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a98395ed74c4c2eb265a141aec14bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.629627 16401 tablet_service.cc:1467] Processing CreateTablet for tablet eb27b3bb29754d7399b0314ada6b1ba7 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ef46a85b1ca942a9ae1f3a3aad6af281]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.630044 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb27b3bb29754d7399b0314ada6b1ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.634799 16135 tablet_service.cc:1467] Processing CreateTablet for tablet eb27b3bb29754d7399b0314ada6b1ba7 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ef46a85b1ca942a9ae1f3a3aad6af281]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.635264 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb27b3bb29754d7399b0314ada6b1ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.613515 16002 tablet_service.cc:1467] Processing CreateTablet for tablet 8a98395ed74c4c2eb265a141aec14bd3 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=ef46a85b1ca942a9ae1f3a3aad6af281]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:56.636176 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a98395ed74c4c2eb265a141aec14bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.652535 16881 consensus_queue.cc:1035] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:56.654364 17460 tablet_bootstrap.cc:492] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:56.656780 17460 tablet_bootstrap.cc:654] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.665557 17460 tablet_bootstrap.cc:492] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:56.665777 17460 ts_tablet_manager.cc:1367] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.006s
I20240502 14:14:56.666687 17460 raft_consensus.cc:348] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.666946 17460 raft_consensus.cc:374] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.667071 17460 raft_consensus.cc:729] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.667347 17460 consensus_queue.cc:260] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.668026 17460 ts_tablet_manager.cc:1398] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:56.668298 17460 tablet_bootstrap.cc:492] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:56.669601 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:56.670230 17460 tablet_bootstrap.cc:654] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.680915 17396 tablet.cc:1722] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:56.688917 17397 tablet.cc:1722] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:56.690804 17464 tablet_bootstrap.cc:492] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:56.692387 17464 tablet_bootstrap.cc:654] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.693415 17400 tablet.cc:1722] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:56.694200 17466 tablet_bootstrap.cc:492] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:56.694566 17340 tablet_copy_client.cc:308] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:36687
I20240502 14:14:56.696244 13208 tablet_copy_service.cc:140] P 20a967bad5504b1e9c6fb720ecdd1625: Received BeginTabletCopySession request for tablet f9c268e5a8e7462ab8e940775c73e591 from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:48115)
I20240502 14:14:56.696482 13208 tablet_copy_service.cc:161] P 20a967bad5504b1e9c6fb720ecdd1625: Beginning new tablet copy session on tablet f9c268e5a8e7462ab8e940775c73e591 from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:48115: session id = 5297e816f782430d9a01220f0471a882-f9c268e5a8e7462ab8e940775c73e591
I20240502 14:14:56.697213 17466 tablet_bootstrap.cc:654] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.700503 13208 tablet_copy_source_session.cc:205] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:56.701819 17340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9c268e5a8e7462ab8e940775c73e591. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.703416 17458 tablet_bootstrap.cc:492] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:56.704797 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=86de893985484748b7afe6e6f1de6b2c] alter complete (version 3)
I20240502 14:14:56.707152 17458 tablet_bootstrap.cc:654] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.708953 17340 tablet_copy_client.cc:783] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:56.708978 17464 tablet_bootstrap.cc:492] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:56.709116 17464 ts_tablet_manager.cc:1367] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240502 14:14:56.709190 17340 tablet_copy_client.cc:647] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:56.710402 17464 raft_consensus.cc:348] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.710773 17464 raft_consensus.cc:374] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.710909 17464 raft_consensus.cc:729] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.711176 17340 tablet_copy_client.cc:523] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:56.711150 17464 consensus_queue.cc:260] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.712502 17464 ts_tablet_manager.cc:1398] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:14:56.712782 17464 tablet_bootstrap.cc:492] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:56.714251 17340 tablet_bootstrap.cc:492] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:56.715273 17464 tablet_bootstrap.cc:654] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.718879 17458 tablet_bootstrap.cc:492] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:56.719034 17458 ts_tablet_manager.cc:1367] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.002s
I20240502 14:14:56.719856 17458 raft_consensus.cc:348] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.720124 17458 raft_consensus.cc:374] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.720232 17458 raft_consensus.cc:729] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.720448 17458 consensus_queue.cc:260] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.721014 17458 ts_tablet_manager.cc:1398] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:56.722334 17459 tablet_bootstrap.cc:492] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:56.723172 17464 tablet_bootstrap.cc:492] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:56.723714 17464 ts_tablet_manager.cc:1367] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240502 14:14:56.724488 17464 raft_consensus.cc:348] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:56.724623 17459 tablet_bootstrap.cc:654] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.726159 17464 raft_consensus.cc:374] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.726379 17464 raft_consensus.cc:729] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.726780 17464 consensus_queue.cc:260] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:56.727905 17464 ts_tablet_manager.cc:1398] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:14:56.731333 17466 tablet_bootstrap.cc:492] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:56.731485 17466 ts_tablet_manager.cc:1367] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.003s
I20240502 14:14:56.732290 17466 raft_consensus.cc:348] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.732550 17466 raft_consensus.cc:374] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.732674 17466 raft_consensus.cc:729] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.732981 17466 consensus_queue.cc:260] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.734187 17460 tablet_bootstrap.cc:492] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:56.734350 17460 ts_tablet_manager.cc:1367] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.066s user 0.020s sys 0.022s
I20240502 14:14:56.735077 17460 raft_consensus.cc:348] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:56.735298 17460 raft_consensus.cc:374] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.735399 17460 raft_consensus.cc:729] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.735749 17340 tablet_bootstrap.cc:492] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:56.735698 17460 consensus_queue.cc:260] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:56.736107 17340 tablet_bootstrap.cc:492] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882: Bootstrap complete.
I20240502 14:14:56.736248 17460 ts_tablet_manager.cc:1398] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:56.736346 17340 ts_tablet_manager.cc:1367] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20240502 14:14:56.736400 17466 ts_tablet_manager.cc:1398] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:14:56.737113 17340 raft_consensus.cc:348] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:56.737383 17340 raft_consensus.cc:729] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: LEARNER
I20240502 14:14:56.737565 17340 consensus_queue.cc:260] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:56.738224 17340 ts_tablet_manager.cc:1398] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:14:56.739290 13208 tablet_copy_service.cc:342] P 20a967bad5504b1e9c6fb720ecdd1625: Request end of tablet copy session 5297e816f782430d9a01220f0471a882-f9c268e5a8e7462ab8e940775c73e591 received from {username='slave'} at 127.6.196.199:48115
I20240502 14:14:56.739403 13208 tablet_copy_service.cc:433] P 20a967bad5504b1e9c6fb720ecdd1625: ending tablet copy session 5297e816f782430d9a01220f0471a882-f9c268e5a8e7462ab8e940775c73e591 on tablet f9c268e5a8e7462ab8e940775c73e591 with peer 5297e816f782430d9a01220f0471a882
I20240502 14:14:56.741655 17459 tablet_bootstrap.cc:492] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:56.741837 17459 ts_tablet_manager.cc:1367] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20240502 14:14:56.742635 17459 raft_consensus.cc:348] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.742897 17459 raft_consensus.cc:374] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.743021 17459 raft_consensus.cc:729] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.743330 17459 consensus_queue.cc:260] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.745590 17459 ts_tablet_manager.cc:1398] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:14:56.745877 17459 tablet_bootstrap.cc:492] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:56.747481 17459 tablet_bootstrap.cc:654] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.750433 17459 tablet_bootstrap.cc:492] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:56.750622 17459 ts_tablet_manager.cc:1367] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:14:56.751545 17459 raft_consensus.cc:348] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.751837 17459 raft_consensus.cc:374] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.751946 17459 raft_consensus.cc:729] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.752264 17459 consensus_queue.cc:260] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.752907 17459 ts_tablet_manager.cc:1398] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 14:14:56.753264 17459 tablet_bootstrap.cc:492] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:56.755025 17459 tablet_bootstrap.cc:654] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.757633 17318 ts_tablet_manager.cc:902] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:56.763228 17459 tablet_bootstrap.cc:492] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:56.763351 17459 ts_tablet_manager.cc:1367] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20240502 14:14:56.763855 17459 raft_consensus.cc:348] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:56.763981 17459 raft_consensus.cc:374] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:56.764135 17459 raft_consensus.cc:729] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:56.764403 17459 consensus_queue.cc:260] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:56.764931 17459 ts_tablet_manager.cc:1398] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20240502 14:14:56.771764 17086 raft_consensus.cc:1050] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:14:56.772790 17086 consensus_queue.cc:237] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:14:56.779043 16419 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:56.779845 13188 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:56.779821 17082 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:56.780962 17083 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:56.782025 13315 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:56.786734 17083 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:56.787427 17128 raft_consensus.cc:482] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:56.787648 17128 raft_consensus.cc:504] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.788456 17128 leader_election.cc:288] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:56.790454 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb27b3bb29754d7399b0314ada6b1ba7" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:14:56.790688 16021 raft_consensus.cc:2454] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:14:56.791216 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb27b3bb29754d7399b0314ada6b1ba7" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:56.791455 16420 raft_consensus.cc:2454] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:14:56.791993 16090 leader_election.cc:302] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:56.794027 17128 raft_consensus.cc:2781] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:56.794160 17128 raft_consensus.cc:482] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:56.794276 17128 raft_consensus.cc:3037] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.799082 17086 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:56.801849 13188 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:56.807901 13044 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:56.808171 13044 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:56.812589 13044 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 798f5ac2da8d4172b154199bcd57d75d
I20240502 14:14:56.814090 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet 798f5ac2da8d4172b154199bcd57d75d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:56 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:56.815752 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 798f5ac2da8d4172b154199bcd57d75d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:56 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:56.817993 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet 798f5ac2da8d4172b154199bcd57d75d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:56 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:14:56.821403 16929 raft_consensus.cc:482] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:56.821599 16929 raft_consensus.cc:504] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.822469 16929 leader_election.cc:288] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:56.823637 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "819e6e83e6064c539cfc73a28c43db5b" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:14:56.823915 16021 raft_consensus.cc:2454] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:56.824398 13253 leader_election.cc:302] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:56.824849 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "819e6e83e6064c539cfc73a28c43db5b" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:14:56.825116 16284 raft_consensus.cc:2454] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:14:56.825870 16929 raft_consensus.cc:2781] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:56.826014 16929 raft_consensus.cc:482] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:56.826120 16929 raft_consensus.cc:3037] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.830840 16929 raft_consensus.cc:504] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.831949 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "819e6e83e6064c539cfc73a28c43db5b" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:14:56.832193 16284 raft_consensus.cc:3037] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.835458 16284 raft_consensus.cc:2454] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:56.835475 17128 raft_consensus.cc:504] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.836409 13253 leader_election.cc:302] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:14:56.836771 16881 raft_consensus.cc:2781] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:56.836722 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb27b3bb29754d7399b0314ada6b1ba7" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:14:56.836972 16881 raft_consensus.cc:686] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:14:56.836985 16420 raft_consensus.cc:3037] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.837318 16881 consensus_queue.cc:237] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.840740 16420 raft_consensus.cc:2454] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:14:56.841660 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "819e6e83e6064c539cfc73a28c43db5b" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:14:56.841933 16021 raft_consensus.cc:3037] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.845675 16021 raft_consensus.cc:2454] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:14:56.845960 16089 leader_election.cc:302] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:56.846494 16929 leader_election.cc:288] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:56.846966 16804 raft_consensus.cc:2781] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:56.847153 16804 raft_consensus.cc:686] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:14:56.847460 16804 consensus_queue.cc:237] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:56.849678 17399 raft_consensus.cc:1050] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER c86f7440570f4796babfd44c3d7fd847 to VOTER
I20240502 14:14:56.850948 17399 consensus_queue.cc:237] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:14:56.872387 17128 leader_election.cc:288] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:56.872753 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb27b3bb29754d7399b0314ada6b1ba7" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:14:56.873036 16021 raft_consensus.cc:3037] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:56.872921 13040 catalog_manager.cc:5484] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:56.876855 16021 raft_consensus.cc:2454] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:14:56.886559 13315 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:56.898130 13040 catalog_manager.cc:5484] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:56.924552 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:56.928287 17491 tablet_replica.cc:330] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:14:56.928596 17491 raft_consensus.cc:2229] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:56.928889 17491 raft_consensus.cc:2258] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:56.929570 17489 tablet_replica.cc:330] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:56.929841 17489 raft_consensus.cc:2229] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:56.930339 17489 raft_consensus.cc:2258] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:56.931418 17489 ts_tablet_manager.cc:1875] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:56.937192 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 16449b78bf4b46d4aa9b919260d9417f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=61947a44845d4258833ec80cdf1dac21]), partition=
I20240502 14:14:56.937357 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 16449b78bf4b46d4aa9b919260d9417f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=61947a44845d4258833ec80cdf1dac21]), partition=
I20240502 14:14:56.937701 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16449b78bf4b46d4aa9b919260d9417f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.937881 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16449b78bf4b46d4aa9b919260d9417f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.941704 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 16449b78bf4b46d4aa9b919260d9417f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=61947a44845d4258833ec80cdf1dac21]), partition=
I20240502 14:14:56.942242 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16449b78bf4b46d4aa9b919260d9417f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:56.947856 16420 raft_consensus.cc:1203] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:56.952327 17459 tablet_bootstrap.cc:492] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:56.954113 17459 tablet_bootstrap.cc:654] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:56.957921 17491 ts_tablet_manager.cc:1875] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:56.961287 16881 raft_consensus.cc:1050] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER d1da7f11f0604f78934354a992bb0f55 to VOTER
I20240502 14:14:56.962438 16881 consensus_queue.cc:237] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:14:56.965035 16023 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:56.966296 13580 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:56.966935 17447 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:56.968806 13446 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:56.969588 16929 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:56.972685 16929 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:56.977886 16929 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:56.981498 13580 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:56.989118 16284 raft_consensus.cc:1203] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:56.989245 16023 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:56.997642 13446 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.005756 13046 catalog_manager.cc:5484] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.011463 17490 tablet_replica.cc:330] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:57.011797 17490 raft_consensus.cc:2229] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:57.012181 17490 raft_consensus.cc:2258] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:57.015604 16284 raft_consensus.cc:1261] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:57.015705 13446 raft_consensus.cc:1261] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:57.016449 17398 consensus_queue.cc:1035] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:57.017108 17398 consensus_queue.cc:1035] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:57.029825 17490 ts_tablet_manager.cc:1875] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:57.030311 16420 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:57.034889 17500 tablet_bootstrap.cc:492] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:14:57.042886 17500 tablet_bootstrap.cc:654] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.045902 13447 consensus_queue.cc:237] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:14:57.047735 17459 tablet_bootstrap.cc:492] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:57.048002 17459 ts_tablet_manager.cc:1367] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.096s user 0.020s sys 0.020s
I20240502 14:14:57.048841 17459 raft_consensus.cc:348] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:57.049213 17459 raft_consensus.cc:374] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.049188 13315 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:57.049391 17459 raft_consensus.cc:729] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.049700 17459 consensus_queue.cc:260] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:57.050472 17459 ts_tablet_manager.cc:1398] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:57.052587 17086 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:57.056316 17086 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:57.058434 17445 raft_consensus.cc:482] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:57.058569 17445 raft_consensus.cc:504] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:57.059307 17445 leader_election.cc:288] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:57.059379 17500 tablet_bootstrap.cc:492] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:14:57.059546 17500 ts_tablet_manager.cc:1367] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240502 14:14:57.060173 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a98395ed74c4c2eb265a141aec14bd3" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:57.060415 16419 raft_consensus.cc:2454] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:14:57.060950 15956 leader_election.cc:302] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:57.061347 17445 raft_consensus.cc:2781] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:57.061458 17445 raft_consensus.cc:482] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:57.061535 17445 raft_consensus.cc:3037] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.063081 13315 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:57.063925 17445 raft_consensus.cc:504] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:57.064148 17500 raft_consensus.cc:348] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:57.064426 17500 raft_consensus.cc:374] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.064522 17500 raft_consensus.cc:729] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.065105 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a98395ed74c4c2eb265a141aec14bd3" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:14:57.065304 16419 raft_consensus.cc:3037] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.067620 17490 ts_tablet_manager.cc:1888] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:57.067836 17490 log.cc:1196] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/798f5ac2da8d4172b154199bcd57d75d
I20240502 14:14:57.068220 16419 raft_consensus.cc:2454] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:14:57.068379 17490 ts_tablet_manager.cc:1909] T 798f5ac2da8d4172b154199bcd57d75d P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:14:57.068699 15956 leader_election.cc:302] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:57.069114 17445 leader_election.cc:288] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:57.069461 17445 raft_consensus.cc:2781] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:57.069806 17445 raft_consensus.cc:686] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:14:57.070257 17445 consensus_queue.cc:237] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:57.070816 17489 ts_tablet_manager.cc:1888] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:57.071208 17500 consensus_queue.cc:260] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:57.071995 17500 ts_tablet_manager.cc:1398] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:14:57.075409 16420 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:57.076958 17082 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:57.078817 16419 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:57.082213 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3e9160c46d64fa0a964cc39aec91dfd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:57.086638 17460 tablet_bootstrap.cc:492] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:57.090054 16155 raft_consensus.cc:1261] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:57.091648 17399 consensus_queue.cc:1035] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:57.091768 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a98395ed74c4c2eb265a141aec14bd3" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:57.094434 16154 raft_consensus.cc:2454] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:14:57.095964 13036 catalog_manager.cc:5484] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:57.080272 13040 catalog_manager.cc:5484] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:57.105049 17489 log.cc:1196] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/798f5ac2da8d4172b154199bcd57d75d
I20240502 14:14:57.105573 17489 ts_tablet_manager.cc:1909] T 798f5ac2da8d4172b154199bcd57d75d P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:57.106709 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 798f5ac2da8d4172b154199bcd57d75d (table rebalancer_dynamic_table_new_name [id=86de893985484748b7afe6e6f1de6b2c]) successfully deleted
I20240502 14:14:57.111975 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 798f5ac2da8d4172b154199bcd57d75d (table rebalancer_dynamic_table_new_name [id=86de893985484748b7afe6e6f1de6b2c]) successfully deleted
I20240502 14:14:57.114174 16881 raft_consensus.cc:1050] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 5297e816f782430d9a01220f0471a882 to VOTER
I20240502 14:14:57.115470 16881 consensus_queue.cc:237] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:14:57.117636 17460 tablet_bootstrap.cc:654] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.118645 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet c3e9160c46d64fa0a964cc39aec91dfd with delete_type TABLET_DATA_TOMBSTONED (TS 20a967bad5504b1e9c6fb720ecdd1625 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:57.118999 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a98395ed74c4c2eb265a141aec14bd3" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:57.119223 16155 raft_consensus.cc:3037] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.127948 16155 raft_consensus.cc:2454] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:14:57.130784 16288 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:57.131772 17460 tablet_bootstrap.cc:492] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:57.131925 17460 ts_tablet_manager.cc:1367] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.001s
I20240502 14:14:57.132139 17398 raft_consensus.cc:2932] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.132776 17460 raft_consensus.cc:348] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:57.133136 17460 raft_consensus.cc:374] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.133316 17460 raft_consensus.cc:729] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.133562 17460 consensus_queue.cc:260] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:57.133812 13579 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:57.134413 17460 ts_tablet_manager.cc:1398] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:14:57.134680 13045 catalog_manager.cc:5484] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:57.139690 16929 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:57.140412 16929 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:57.144848 17318 tablet_copy_client.cc:308] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:14:57.145520 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet 652e49046212458280095c016a342f47 from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:42449)
I20240502 14:14:57.145644 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet 652e49046212458280095c016a342f47 from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:42449: session id = c86f7440570f4796babfd44c3d7fd847-652e49046212458280095c016a342f47
I20240502 14:14:57.147576 16154 raft_consensus.cc:2932] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.148303 13601 tablet_copy_source_session.cc:205] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:57.149966 17318 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652e49046212458280095c016a342f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.154007 17453 raft_consensus.cc:1050] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625: attempting to promote NON_VOTER d1da7f11f0604f78934354a992bb0f55 to VOTER
I20240502 14:14:57.155241 17453 consensus_queue.cc:237] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:14:57.159252 13446 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:57.159296 13315 consensus_queue.cc:237] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:14:57.160914 16021 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:57.161484 17447 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:57.163748 17514 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.164566 16929 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:57.166424 17082 raft_consensus.cc:1050] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER c86f7440570f4796babfd44c3d7fd847 to VOTER
I20240502 14:14:57.167244 17082 consensus_queue.cc:237] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:14:57.169275 16021 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.171427 17318 tablet_copy_client.cc:783] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:57.171613 17318 tablet_copy_client.cc:647] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:57.171851 13188 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:57.172359 17082 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:57.172485 13319 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:57.172981 17082 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:57.175979 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a6b71f7e77b4084a8904b1045099472 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:57.177809 13044 catalog_manager.cc:5484] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:57.178246 17318 tablet_copy_client.cc:523] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:57.180717 13578 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:57.182904 17447 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:57.184691 13447 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:57.185088 16021 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:57.185197 17453 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:57.185580 13319 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:57.185716 17057 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:57.186370 13578 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.188717 17453 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:57.195736 13445 raft_consensus.cc:2932] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.202893 13042 catalog_manager.cc:5484] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.203707 17086 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.208009 17490 tablet_replica.cc:330] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:57.209465 17490 raft_consensus.cc:2229] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:57.209944 17490 raft_consensus.cc:2258] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:57.209698 13315 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.210951 16155 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:57.211553 17083 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:57.214270 13186 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.218498 17057 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.219815 13042 catalog_manager.cc:5484] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:57.221894 13447 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.227721 16023 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.229349 13315 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.235471 13040 catalog_manager.cc:5484] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.238767 17491 ts_tablet_manager.cc:1888] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:57.238927 17491 log.cc:1196] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/798f5ac2da8d4172b154199bcd57d75d
I20240502 14:14:57.239403 17491 ts_tablet_manager.cc:1909] T 798f5ac2da8d4172b154199bcd57d75d P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:14:57.242264 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6b71f7e77b4084a8904b1045099472 with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:57.245092 13580 consensus_queue.cc:237] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:14:57.247435 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 798f5ac2da8d4172b154199bcd57d75d (table rebalancer_dynamic_table_new_name [id=86de893985484748b7afe6e6f1de6b2c]) successfully deleted
I20240502 14:14:57.255118 16153 raft_consensus.cc:1261] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:57.256042 17399 consensus_queue.cc:1035] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:57.257736 16881 consensus_queue.cc:1035] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.260164 17528 tablet_replica.cc:330] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:57.260546 17528 raft_consensus.cc:2229] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:57.260852 17528 raft_consensus.cc:2258] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:57.260886 17229 raft_consensus.cc:2932] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.261977 17528 ts_tablet_manager.cc:1875] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:57.266151 16881 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.267050 16153 raft_consensus.cc:2932] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.270156 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 16d6faae2add42c5bd4af6018a13b841 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:57.272941 13578 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.276861 13446 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.282074 13042 catalog_manager.cc:5484] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.287474 13186 consensus_queue.cc:237] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:14:57.289757 16021 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:57.290292 17490 ts_tablet_manager.cc:1875] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:57.290470 17371 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:57.290648 13315 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:57.291508 13447 raft_consensus.cc:1261] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:57.291913 17371 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:57.292083 17399 consensus_queue.cc:1035] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:57.294119 17371 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.294759 16881 consensus_queue.cc:1035] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.280978 13044 catalog_manager.cc:5484] T 16d6faae2add42c5bd4af6018a13b841 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.297561 13315 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.300879 13446 raft_consensus.cc:2932] T 16d6faae2add42c5bd4af6018a13b841 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.303318 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b2905b7940fb45c49ef8b18943dae256 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:57.303866 13042 catalog_manager.cc:5484] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.304967 16021 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:14:57.326268 17528 ts_tablet_manager.cc:1888] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:14:57.326270 16265 tablet_service.cc:1514] Processing DeleteTablet for tablet 16d6faae2add42c5bd4af6018a13b841 with delete_type TABLET_DATA_TOMBSTONED (TS 5297e816f782430d9a01220f0471a882 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42272
I20240502 14:14:57.327246 17528 log.cc:1196] T 8a6b71f7e77b4084a8904b1045099472 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/8a6b71f7e77b4084a8904b1045099472
I20240502 14:14:57.330298 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 8a6b71f7e77b4084a8904b1045099472 (table rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]) successfully deleted
I20240502 14:14:57.330914 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet b2905b7940fb45c49ef8b18943dae256 with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:57.333513 17528 tablet_replica.cc:330] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:57.333854 17528 raft_consensus.cc:2229] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:57.334156 17528 raft_consensus.cc:2258] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:57.334381 16155 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.339164 16288 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.343298 17528 ts_tablet_manager.cc:1875] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:57.346364 17538 tablet_replica.cc:330] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:14:57.346589 17538 raft_consensus.cc:2229] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:57.346810 17538 pending_rounds.cc:62] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882: Trying to abort 1 pending ops.
I20240502 14:14:57.346926 17538 pending_rounds.cc:68] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023244479898624000 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "16d6faae2add42c5bd4af6018a13b841" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } } }
I20240502 14:14:57.347447 17538 raft_consensus.cc:2894] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 14:14:57.347568 17538 raft_consensus.cc:2866] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:14:57.347683 17538 raft_consensus.cc:2258] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:57.351342 13315 consensus_queue.cc:237] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:14:57.351579 17538 ts_tablet_manager.cc:1875] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:57.356235 13446 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:57.357075 17447 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:57.358942 17490 ts_tablet_manager.cc:1888] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:14:57.359117 17490 log.cc:1196] T c3e9160c46d64fa0a964cc39aec91dfd P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/c3e9160c46d64fa0a964cc39aec91dfd
I20240502 14:14:57.359293 17528 ts_tablet_manager.cc:1888] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:14:57.359493 17528 log.cc:1196] T b2905b7940fb45c49ef8b18943dae256 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/b2905b7940fb45c49ef8b18943dae256
I20240502 14:14:57.359928 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet c3e9160c46d64fa0a964cc39aec91dfd (table rebalance_test_table_17 [id=cbc2cd67f3c24195825b542cf2878195]) successfully deleted
I20240502 14:14:57.359715 17447 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.360412 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet b2905b7940fb45c49ef8b18943dae256 (table rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]) successfully deleted
I20240502 14:14:57.362280 17318 tablet_bootstrap.cc:492] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:57.363003 13446 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.363956 17453 raft_consensus.cc:1050] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625: attempting to promote NON_VOTER 5297e816f782430d9a01220f0471a882 to VOTER
I20240502 14:14:57.364872 17453 consensus_queue.cc:237] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:14:57.367131 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9a95b9049a344b98be91b9961f3e5bee with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:57.369428 13046 catalog_manager.cc:5484] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 5 to 6, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:57.372702 17538 ts_tablet_manager.cc:1888] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:14:57.376305 17538 log.cc:1196] T 16d6faae2add42c5bd4af6018a13b841 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/16d6faae2add42c5bd4af6018a13b841
I20240502 14:14:57.377277 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 16d6faae2add42c5bd4af6018a13b841 (table rebalancer_extra_table_4 [id=cb136ab1e407401daef8e9856e8ee6f9]) successfully deleted
I20240502 14:14:57.381376 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a95b9049a344b98be91b9961f3e5bee with delete_type TABLET_DATA_TOMBSTONED (TS 7eb4ecb2379b44c0954f2658c4a04350 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:57.382349 17128 consensus_queue.cc:1035] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.403051 17229 raft_consensus.cc:482] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:57.403199 17229 raft_consensus.cc:504] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:57.403885 17229 leader_election.cc:288] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:57.404629 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16449b78bf4b46d4aa9b919260d9417f" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:14:57.404861 16022 raft_consensus.cc:2454] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:57.405347 13516 leader_election.cc:302] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:57.405869 17398 raft_consensus.cc:2781] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:57.405961 17398 raft_consensus.cc:482] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:57.406025 17398 raft_consensus.cc:3037] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.408814 17398 raft_consensus.cc:504] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:57.410879 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16449b78bf4b46d4aa9b919260d9417f" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:57.411160 16155 raft_consensus.cc:2454] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:14:57.412066 17398 leader_election.cc:288] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:57.412832 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16449b78bf4b46d4aa9b919260d9417f" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:14:57.412997 16022 raft_consensus.cc:3037] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.415674 16022 raft_consensus.cc:2454] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:57.416152 13516 leader_election.cc:302] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:57.416455 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16449b78bf4b46d4aa9b919260d9417f" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:57.416620 16155 raft_consensus.cc:3037] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.420156 16155 raft_consensus.cc:2454] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:14:57.420889 17398 raft_consensus.cc:2781] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:57.421536 17398 raft_consensus.cc:686] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:14:57.422127 17398 consensus_queue.cc:237] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:14:57.433559 13046 catalog_manager.cc:5484] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:57.434876 17544 tablet_replica.cc:330] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:57.435125 17544 raft_consensus.cc:2229] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:57.435532 17544 raft_consensus.cc:2258] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:57.439203 16285 raft_consensus.cc:1261] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:57.440032 17057 consensus_queue.cc:1035] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:57.444195 17544 ts_tablet_manager.cc:1875] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:57.449316 16419 raft_consensus.cc:1261] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:57.449942 17057 consensus_queue.cc:1035] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:57.452842 13580 raft_consensus.cc:1261] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:57.460642 17453 consensus_queue.cc:1035] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20240502 14:14:57.461474 16804 consensus_queue.cc:1035] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.476425 16284 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:57.477366 17447 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:57.478602 16284 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.490293 17507 consensus_queue.cc:1035] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.506459 17057 raft_consensus.cc:2932] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.511922 17519 raft_consensus.cc:1050] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:14:57.512725 17519 consensus_queue.cc:237] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:14:57.517321 13579 raft_consensus.cc:2932] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.521680 13046 catalog_manager.cc:5484] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 2 to 3, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:57.526441 17544 ts_tablet_manager.cc:1888] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:14:57.526597 17544 log.cc:1196] T 9a95b9049a344b98be91b9961f3e5bee P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/9a95b9049a344b98be91b9961f3e5bee
I20240502 14:14:57.527563 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 9a95b9049a344b98be91b9961f3e5bee (table rebalance_test_table_15 [id=557f9d603bda4eb385d26221f3612c33]) successfully deleted
I20240502 14:14:57.534258 16287 raft_consensus.cc:2932] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.532919 16420 raft_consensus.cc:2932] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.544529 13188 consensus_queue.cc:237] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:14:57.547555 16287 raft_consensus.cc:1261] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:57.548192 17057 consensus_queue.cc:1035] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:57.550794 17371 raft_consensus.cc:2932] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.553006 13319 raft_consensus.cc:1261] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:57.553532 17086 consensus_queue.cc:1035] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:57.554425 16287 raft_consensus.cc:2932] T f9c268e5a8e7462ab8e940775c73e591 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.561087 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f9c268e5a8e7462ab8e940775c73e591 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:57.564296 13044 catalog_manager.cc:5484] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 3 to 4, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:57.565574 13188 raft_consensus.cc:1261] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:57.570111 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet f9c268e5a8e7462ab8e940775c73e591 with delete_type TABLET_DATA_TOMBSTONED (TS 7eb4ecb2379b44c0954f2658c4a04350 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:57.570595 17544 tablet_replica.cc:330] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:57.570850 17544 raft_consensus.cc:2229] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:57.571125 17544 raft_consensus.cc:2258] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:57.571943 17544 ts_tablet_manager.cc:1875] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:57.572283 13046 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:57.579517 16153 raft_consensus.cc:1261] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:57.580085 17086 consensus_queue.cc:1035] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:57.580508 17398 consensus_queue.cc:1035] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.589010 16022 raft_consensus.cc:1261] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:57.589815 17398 consensus_queue.cc:1035] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.591271 17544 ts_tablet_manager.cc:1888] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:14:57.594395 13042 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 16449b78bf4b46d4aa9b919260d9417f (table rebalancer_dynamic_table [id=61947a44845d4258833ec80cdf1dac21]). Expected version 1 got 0
I20240502 14:14:57.601136 13042 catalog_manager.cc:5576] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817) does not have the latest schema for tablet 16449b78bf4b46d4aa9b919260d9417f (table rebalancer_dynamic_table [id=61947a44845d4258833ec80cdf1dac21]). Expected version 1 got 0
I20240502 14:14:57.598412 17565 tablet.cc:1722] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:57.602497 17565 tablet.cc:1620] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: MemRowSet was empty: no flush needed.
I20240502 14:14:57.605353 17544 log.cc:1196] T f9c268e5a8e7462ab8e940775c73e591 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/f9c268e5a8e7462ab8e940775c73e591
I20240502 14:14:57.606266 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet f9c268e5a8e7462ab8e940775c73e591 (table rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]) successfully deleted
I20240502 14:14:57.606402 13042 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 16449b78bf4b46d4aa9b919260d9417f (table rebalancer_dynamic_table [id=61947a44845d4258833ec80cdf1dac21]). Expected version 1 got 0
I20240502 14:14:57.601680 17566 tablet.cc:1722] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:57.610112 17566 tablet.cc:1620] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:57.618975 16419 raft_consensus.cc:1261] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:57.620440 17086 consensus_queue.cc:1035] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:57.623469 16421 raft_consensus.cc:1261] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:57.629339 17453 consensus_queue.cc:1035] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:57.635900 17086 raft_consensus.cc:2932] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:57.636296 13188 raft_consensus.cc:2932] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:57.643337 13315 raft_consensus.cc:2932] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:57.646672 13046 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=61947a44845d4258833ec80cdf1dac21] alter complete (version 1)
I20240502 14:14:57.650157 13045 catalog_manager.cc:5484] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:57.651749 17563 consensus_queue.cc:1035] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:57.652645 16420 raft_consensus.cc:2932] T f9c268e5a8e7462ab8e940775c73e591 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:57.665290 16419 raft_consensus.cc:2932] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:57.671466 13446 consensus_queue.cc:237] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:14:57.674432 13315 raft_consensus.cc:1261] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:57.675268 13044 catalog_manager.cc:5576] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695) does not have the latest schema for tablet 16449b78bf4b46d4aa9b919260d9417f (table rebalancer_dynamic_table [id=61947a44845d4258833ec80cdf1dac21]). Expected version 1 got 0
I20240502 14:14:57.675244 17083 consensus_queue.cc:1035] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:57.678956 17086 raft_consensus.cc:2932] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:57.681767 13317 raft_consensus.cc:2932] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:57.684592 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet abd272455c064ba48726d933d53c83ed with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:57.687469 17577 tablet.cc:1722] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:57.688915 13046 catalog_manager.cc:5484] T abd272455c064ba48726d933d53c83ed P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:57.690322 17577 tablet.cc:1620] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:14:57.694993 16420 raft_consensus.cc:1261] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:57.695798 17086 consensus_queue.cc:1035] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:57.697299 13446 consensus_queue.cc:237] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:14:57.697392 16420 raft_consensus.cc:2932] T abd272455c064ba48726d933d53c83ed P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:57.701361 13188 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:57.702251 17086 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:57.702340 16155 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:57.703260 17082 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:57.704313 17318 tablet_bootstrap.cc:492] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:57.704766 17318 tablet_bootstrap.cc:492] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:14:57.705018 17318 ts_tablet_manager.cc:1367] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.343s user 0.048s sys 0.012s
I20240502 14:14:57.704792 17082 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.705192 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet abd272455c064ba48726d933d53c83ed with delete_type TABLET_DATA_TOMBSTONED (TS 20a967bad5504b1e9c6fb720ecdd1625 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37690
I20240502 14:14:57.705547 17490 tablet_replica.cc:330] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:14:57.705783 17490 raft_consensus.cc:2229] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:57.706180 17490 raft_consensus.cc:2258] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:57.707091 17490 ts_tablet_manager.cc:1875] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:57.710610 17318 raft_consensus.cc:348] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:57.710906 17318 raft_consensus.cc:729] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:14:57.711117 17318 consensus_queue.cc:260] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:57.711685 13188 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.712320 17318 ts_tablet_manager.cc:1398] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:14:57.714493 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-652e49046212458280095c016a342f47 received from {username='slave'} at 127.6.196.198:42449
I20240502 14:14:57.714612 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-652e49046212458280095c016a342f47 on tablet 652e49046212458280095c016a342f47 with peer c86f7440570f4796babfd44c3d7fd847
I20240502 14:14:57.715626 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7852539de07744caaf385620c5166861 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:57.716454 16155 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:57.719601 13045 catalog_manager.cc:5484] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:57.724936 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet 7852539de07744caaf385620c5166861 with delete_type TABLET_DATA_TOMBSTONED (TS cd67493f6f0142f896010f2b8a39887d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:57.729470 16155 raft_consensus.cc:1203] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:57.738778 17490 ts_tablet_manager.cc:1888] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:14:57.739097 17490 log.cc:1196] T abd272455c064ba48726d933d53c83ed P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/abd272455c064ba48726d933d53c83ed
I20240502 14:14:57.740356 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet abd272455c064ba48726d933d53c83ed (table rebalance_test_table_11 [id=b7310fe1c01b4fe7b29328321c07e72d]) successfully deleted
I20240502 14:14:57.740447 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:57.744570 17581 tablet_replica.cc:330] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:57.744805 17581 raft_consensus.cc:2229] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:57.745098 17581 raft_consensus.cc:2258] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:57.746029 17581 ts_tablet_manager.cc:1875] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:57.750936 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea4a9b24178462dbe84e3c33f2d2245 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=71087d0bec7e450da33bd69a7f555355]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.751271 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea4a9b24178462dbe84e3c33f2d2245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.751456 16265 tablet_service.cc:1467] Processing CreateTablet for tablet 1b34f2a22306459784a1ad7b81cd1683 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=71087d0bec7e450da33bd69a7f555355]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.751827 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b34f2a22306459784a1ad7b81cd1683. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.753541 17581 ts_tablet_manager.cc:1888] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:14:57.753636 17581 log.cc:1196] T 7852539de07744caaf385620c5166861 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/7852539de07744caaf385620c5166861
I20240502 14:14:57.754784 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 7852539de07744caaf385620c5166861 (table rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]) successfully deleted
I20240502 14:14:57.755432 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea4a9b24178462dbe84e3c33f2d2245 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=71087d0bec7e450da33bd69a7f555355]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.755776 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea4a9b24178462dbe84e3c33f2d2245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.762749 16401 tablet_service.cc:1467] Processing CreateTablet for tablet 1b34f2a22306459784a1ad7b81cd1683 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=71087d0bec7e450da33bd69a7f555355]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.762884 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 1b34f2a22306459784a1ad7b81cd1683 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=71087d0bec7e450da33bd69a7f555355]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.763159 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b34f2a22306459784a1ad7b81cd1683. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.763275 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b34f2a22306459784a1ad7b81cd1683. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.763489 13046 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:57.765489 16133 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd07adcbb1f4bd4bcb13bd089c89974 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=71087d0bec7e450da33bd69a7f555355]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.765828 16133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd07adcbb1f4bd4bcb13bd089c89974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.767778 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 4ea4a9b24178462dbe84e3c33f2d2245 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=71087d0bec7e450da33bd69a7f555355]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.768159 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ea4a9b24178462dbe84e3c33f2d2245. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.773746 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd07adcbb1f4bd4bcb13bd089c89974 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=71087d0bec7e450da33bd69a7f555355]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.774179 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd07adcbb1f4bd4bcb13bd089c89974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.774101 16267 tablet_service.cc:1467] Processing CreateTablet for tablet 0dd07adcbb1f4bd4bcb13bd089c89974 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=71087d0bec7e450da33bd69a7f555355]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:57.774514 16267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd07adcbb1f4bd4bcb13bd089c89974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:57.779712 17585 tablet_bootstrap.cc:492] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:57.779951 17565 tablet.cc:1722] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:57.780485 17565 tablet.cc:1620] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: MemRowSet was empty: no flush needed.
I20240502 14:14:57.780934 17577 tablet.cc:1722] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:57.781414 17577 tablet.cc:1620] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:14:57.784386 17585 tablet_bootstrap.cc:654] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.786635 17566 tablet.cc:1722] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:57.787142 17566 tablet.cc:1620] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:57.791913 17582 tablet_bootstrap.cc:492] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:57.796252 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=61947a44845d4258833ec80cdf1dac21] alter complete (version 2)
I20240502 14:14:57.797737 17582 tablet_bootstrap.cc:654] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.800619 17586 tablet_bootstrap.cc:492] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:57.802738 17586 tablet_bootstrap.cc:654] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.811614 17585 tablet_bootstrap.cc:492] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:57.811985 17585 ts_tablet_manager.cc:1367] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20240502 14:14:57.812816 17585 raft_consensus.cc:348] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:57.814859 17585 raft_consensus.cc:374] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.815300 17585 raft_consensus.cc:729] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.815918 17586 tablet_bootstrap.cc:492] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:14:57.815965 17585 consensus_queue.cc:260] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:57.816123 17586 ts_tablet_manager.cc:1367] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.007s
I20240502 14:14:57.818318 17585 ts_tablet_manager.cc:1398] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:14:57.818936 17589 tablet_bootstrap.cc:492] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:57.822599 17586 raft_consensus.cc:348] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:57.824115 17586 raft_consensus.cc:374] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.824270 17586 raft_consensus.cc:729] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.824522 17586 consensus_queue.cc:260] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:57.825342 17589 tablet_bootstrap.cc:654] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.826246 17586 ts_tablet_manager.cc:1398] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 14:14:57.827363 17582 tablet_bootstrap.cc:492] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:57.827558 17582 ts_tablet_manager.cc:1367] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.002s
I20240502 14:14:57.828248 17582 raft_consensus.cc:348] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:57.828457 17582 raft_consensus.cc:374] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.828549 17582 raft_consensus.cc:729] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.828748 17582 consensus_queue.cc:260] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:57.829330 17582 ts_tablet_manager.cc:1398] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:57.832867 17583 tablet_bootstrap.cc:492] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:57.836191 17583 tablet_bootstrap.cc:654] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.838164 17583 tablet_bootstrap.cc:492] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:57.838294 17583 ts_tablet_manager.cc:1367] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:14:57.838796 17583 raft_consensus.cc:348] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:57.838968 17583 raft_consensus.cc:374] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.839051 17583 raft_consensus.cc:729] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.839226 17583 consensus_queue.cc:260] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:57.839707 17583 ts_tablet_manager.cc:1398] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:14:57.840106 17583 tablet_bootstrap.cc:492] T 4ea4a9b24178462dbe84e3c33f2d2245 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:57.841782 17589 tablet_bootstrap.cc:492] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:57.841933 17589 ts_tablet_manager.cc:1367] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.002s
I20240502 14:14:57.842686 17589 raft_consensus.cc:348] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:57.842864 17589 raft_consensus.cc:374] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.842945 17589 raft_consensus.cc:729] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.843122 17589 consensus_queue.cc:260] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:57.843664 17589 ts_tablet_manager.cc:1398] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:57.843912 17589 tablet_bootstrap.cc:492] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:57.846380 17589 tablet_bootstrap.cc:654] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.850320 17583 tablet_bootstrap.cc:654] T 4ea4a9b24178462dbe84e3c33f2d2245 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.855042 17583 tablet_bootstrap.cc:492] T 4ea4a9b24178462dbe84e3c33f2d2245 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:57.855237 17583 ts_tablet_manager.cc:1367] T 4ea4a9b24178462dbe84e3c33f2d2245 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20240502 14:14:57.855294 17589 tablet_bootstrap.cc:492] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:57.855437 17589 ts_tablet_manager.cc:1367] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.003s
I20240502 14:14:57.855973 17583 raft_consensus.cc:348] T 4ea4a9b24178462dbe84e3c33f2d2245 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:57.856065 17589 raft_consensus.cc:348] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:57.856293 17589 raft_consensus.cc:374] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.856314 17583 raft_consensus.cc:374] T 4ea4a9b24178462dbe84e3c33f2d2245 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.856375 17589 raft_consensus.cc:729] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.856518 17583 raft_consensus.cc:729] T 4ea4a9b24178462dbe84e3c33f2d2245 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.856601 17589 consensus_queue.cc:260] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:57.857127 17589 ts_tablet_manager.cc:1398] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:57.866741 17589 tablet_bootstrap.cc:492] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:57.869424 17589 tablet_bootstrap.cc:654] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.870316 17583 consensus_queue.cc:260] T 4ea4a9b24178462dbe84e3c33f2d2245 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:57.870777 17583 ts_tablet_manager.cc:1398] T 4ea4a9b24178462dbe84e3c33f2d2245 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.015s user 0.000s sys 0.001s
I20240502 14:14:57.871209 17583 tablet_bootstrap.cc:492] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:57.872589 17583 tablet_bootstrap.cc:654] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:57.878113 17167 raft_consensus.cc:482] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:57.878268 17167 raft_consensus.cc:504] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:57.878908 17167 leader_election.cc:288] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:57.879496 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b34f2a22306459784a1ad7b81cd1683" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:14:57.879696 16287 raft_consensus.cc:2454] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:14:57.881115 16355 leader_election.cc:302] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:14:57.881973 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b34f2a22306459784a1ad7b81cd1683" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:57.882189 16155 raft_consensus.cc:2454] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:14:57.883946 17167 raft_consensus.cc:2781] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:57.884104 17167 raft_consensus.cc:482] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:57.884222 17167 raft_consensus.cc:3037] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.885727 17583 tablet_bootstrap.cc:492] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:57.885726 17589 tablet_bootstrap.cc:492] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:57.885854 17583 ts_tablet_manager.cc:1367] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20240502 14:14:57.886080 17589 ts_tablet_manager.cc:1367] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.006s
I20240502 14:14:57.886516 17583 raft_consensus.cc:348] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:57.886684 17583 raft_consensus.cc:374] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.886775 17583 raft_consensus.cc:729] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.886827 17589 raft_consensus.cc:348] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:57.886971 17583 consensus_queue.cc:260] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:57.887079 17589 raft_consensus.cc:374] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:57.887228 17589 raft_consensus.cc:729] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:57.887485 17583 ts_tablet_manager.cc:1398] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:57.887440 17589 consensus_queue.cc:260] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:57.887981 17589 ts_tablet_manager.cc:1398] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:14:57.897425 17167 raft_consensus.cc:504] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:57.898684 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b34f2a22306459784a1ad7b81cd1683" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:14:57.898869 16287 raft_consensus.cc:3037] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.902292 16287 raft_consensus.cc:2454] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:14:57.903616 16355 leader_election.cc:302] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:14:57.909049 13046 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:57.922937 17565 tablet.cc:1722] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:57.924165 17167 leader_election.cc:288] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:57.924681 17167 raft_consensus.cc:2781] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:57.924798 17167 raft_consensus.cc:686] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:14:57.925045 17167 consensus_queue.cc:237] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:57.938211 17566 tablet.cc:1722] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:57.941629 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b34f2a22306459784a1ad7b81cd1683" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:57.941901 16155 raft_consensus.cc:3037] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.946933 16155 raft_consensus.cc:2454] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:14:57.948387 17577 tablet.cc:1722] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:57.961208 13044 catalog_manager.cc:5484] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:57.962560 17128 raft_consensus.cc:482] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:57.962734 17128 raft_consensus.cc:504] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:57.962858 13045 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=61947a44845d4258833ec80cdf1dac21] alter complete (version 3)
I20240502 14:14:57.963387 17128 leader_election.cc:288] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:57.963920 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea4a9b24178462dbe84e3c33f2d2245" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:14:57.964118 16287 raft_consensus.cc:2454] T 4ea4a9b24178462dbe84e3c33f2d2245 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:14:57.964090 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea4a9b24178462dbe84e3c33f2d2245" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:14:57.964309 16021 raft_consensus.cc:2454] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:14:57.964504 16090 leader_election.cc:302] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:57.964761 17128 raft_consensus.cc:2781] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:57.964915 17128 raft_consensus.cc:482] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:57.965008 17128 raft_consensus.cc:3037] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.967089 17128 raft_consensus.cc:504] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:57.967940 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea4a9b24178462dbe84e3c33f2d2245" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:14:57.968029 17128 leader_election.cc:288] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:14:57.968140 16287 raft_consensus.cc:3037] T 4ea4a9b24178462dbe84e3c33f2d2245 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.968124 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ea4a9b24178462dbe84e3c33f2d2245" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:14:57.968276 16021 raft_consensus.cc:3037] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:57.970439 16287 raft_consensus.cc:2454] T 4ea4a9b24178462dbe84e3c33f2d2245 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:14:57.970790 16090 leader_election.cc:302] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:57.970865 16021 raft_consensus.cc:2454] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:14:57.971061 17128 raft_consensus.cc:2781] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:57.971187 17128 raft_consensus.cc:686] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:14:57.971483 17128 consensus_queue.cc:237] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:14:57.974378 13046 catalog_manager.cc:5484] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:58.011042 13317 consensus_queue.cc:237] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:58.013244 13579 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:58.013289 13446 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:58.013612 13253 consensus_peers.cc:485] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0854e9f4851e400f9af18181a71ae8f7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:58.013823 16881 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:58.014077 17447 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:58.016886 16929 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:58.021023 13579 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:58.021194 13446 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:58.027782 13046 catalog_manager.cc:5484] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:58.041532 13579 consensus_queue.cc:237] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:58.043567 13446 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:58.043635 13515 consensus_peers.cc:485] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27857198d4ec430ebee897c90141c58d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:58.043677 13317 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:58.044070 17229 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:58.044673 17229 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:58.046841 17399 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:58.047492 13446 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:58.047786 13317 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:58.051911 13040 catalog_manager.cc:5484] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:58.066640 13446 consensus_queue.cc:237] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:58.069365 13046 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:58.069584 13046 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
W20240502 14:14:58.070021 13384 consensus_peers.cc:485] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab51b049c3674b629b55ed29df5b1fab. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:58.071128 13580 raft_consensus.cc:1261] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:58.071549 13188 raft_consensus.cc:1261] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:58.071590 17086 consensus_queue.cc:1035] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:58.072218 17086 consensus_queue.cc:1035] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:58.072741 13046 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 16449b78bf4b46d4aa9b919260d9417f
I20240502 14:14:58.073465 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet 16449b78bf4b46d4aa9b919260d9417f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:58 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:14:58.074345 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet 16449b78bf4b46d4aa9b919260d9417f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:58 UTC) from {username='slave'} at 127.0.0.1:41222
I20240502 14:14:58.074788 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 16449b78bf4b46d4aa9b919260d9417f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:58 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:14:58.075111 17544 tablet_replica.cc:330] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:14:58.075340 17544 raft_consensus.cc:2229] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:58.075646 17544 raft_consensus.cc:2258] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:58.076754 17544 ts_tablet_manager.cc:1875] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:58.084846 17656 tablet_replica.cc:330] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:14:58.085459 17656 raft_consensus.cc:2229] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:58.086138 17656 raft_consensus.cc:2258] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:58.087671 17654 tablet_replica.cc:330] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:14:58.087849 17654 raft_consensus.cc:2229] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:58.088022 17654 raft_consensus.cc:2258] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:58.089263 17654 ts_tablet_manager.cc:1875] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:58.089290 17083 raft_consensus.cc:2932] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:58.090026 17544 ts_tablet_manager.cc:1888] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:58.090250 17544 log.cc:1196] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/16449b78bf4b46d4aa9b919260d9417f
I20240502 14:14:58.090829 17544 ts_tablet_manager.cc:1909] T 16449b78bf4b46d4aa9b919260d9417f P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:14:58.091841 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 16449b78bf4b46d4aa9b919260d9417f (table rebalancer_dynamic_table_new_name [id=61947a44845d4258833ec80cdf1dac21]) successfully deleted
I20240502 14:14:58.093575 13188 raft_consensus.cc:2932] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:58.094679 17654 ts_tablet_manager.cc:1888] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:58.095391 17654 log.cc:1196] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/16449b78bf4b46d4aa9b919260d9417f
I20240502 14:14:58.096019 17654 ts_tablet_manager.cc:1909] T 16449b78bf4b46d4aa9b919260d9417f P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:14:58.096359 13579 raft_consensus.cc:2932] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:58.097143 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 16449b78bf4b46d4aa9b919260d9417f (table rebalancer_dynamic_table_new_name [id=61947a44845d4258833ec80cdf1dac21]) successfully deleted
I20240502 14:14:58.097255 13046 catalog_manager.cc:5484] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:58.099889 17656 ts_tablet_manager.cc:1875] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:58.106982 17656 ts_tablet_manager.cc:1888] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:58.107612 17656 log.cc:1196] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/16449b78bf4b46d4aa9b919260d9417f
I20240502 14:14:58.108080 17656 ts_tablet_manager.cc:1909] T 16449b78bf4b46d4aa9b919260d9417f P d1da7f11f0604f78934354a992bb0f55: Deleting consensus metadata
I20240502 14:14:58.109141 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet 16449b78bf4b46d4aa9b919260d9417f (table rebalancer_dynamic_table_new_name [id=61947a44845d4258833ec80cdf1dac21]) successfully deleted
I20240502 14:14:58.118186 13580 consensus_queue.cc:237] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:58.121047 13446 raft_consensus.cc:1261] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:58.121047 13188 raft_consensus.cc:1261] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:58.121172 13516 consensus_peers.cc:485] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db06293ca0914bb88dbff3808e3f1874. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:58.121579 17229 consensus_queue.cc:1035] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:58.122089 17398 consensus_queue.cc:1035] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:58.124073 17399 raft_consensus.cc:2932] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:58.124621 13446 raft_consensus.cc:2932] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:58.125367 13188 raft_consensus.cc:2932] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:58.131435 13040 catalog_manager.cc:5484] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:58.150442 13580 consensus_queue.cc:237] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:14:58.152529 16021 raft_consensus.cc:1261] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:14:58.152617 13514 consensus_peers.cc:485] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f5bf499e8c044a1886989a90c6bb69fa. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:58.153013 16418 raft_consensus.cc:1261] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:58.153242 17398 consensus_queue.cc:1035] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:58.153708 17398 consensus_queue.cc:1035] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:58.162721 17399 raft_consensus.cc:2932] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:14:58.163477 16418 raft_consensus.cc:2932] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:14:58.168586 13046 catalog_manager.cc:5484] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:58.183984 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:58.185335 16021 raft_consensus.cc:2932] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:14:58.194224 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 78e08376f3f74f709acda63bb2280575 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff]), partition=
I20240502 14:14:58.194535 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e08376f3f74f709acda63bb2280575. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.194825 16401 tablet_service.cc:1467] Processing CreateTablet for tablet 78e08376f3f74f709acda63bb2280575 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff]), partition=
I20240502 14:14:58.194945 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 78e08376f3f74f709acda63bb2280575 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff]), partition=
I20240502 14:14:58.195135 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e08376f3f74f709acda63bb2280575. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.195334 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78e08376f3f74f709acda63bb2280575. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.197522 17576 raft_consensus.cc:482] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:58.197660 17576 raft_consensus.cc:504] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:58.198426 17576 leader_election.cc:288] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:14:58.198864 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd07adcbb1f4bd4bcb13bd089c89974" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:58.199110 16155 raft_consensus.cc:2454] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:58.199220 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd07adcbb1f4bd4bcb13bd089c89974" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:14:58.199453 16287 raft_consensus.cc:2454] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:14:58.200009 13384 leader_election.cc:302] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:14:58.200481 17576 raft_consensus.cc:2781] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:58.200593 17576 raft_consensus.cc:482] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:58.200670 17576 raft_consensus.cc:3037] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.200932 17582 tablet_bootstrap.cc:492] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:58.202287 17582 tablet_bootstrap.cc:654] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.203197 17576 raft_consensus.cc:504] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:58.204249 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd07adcbb1f4bd4bcb13bd089c89974" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:14:58.204432 16287 raft_consensus.cc:3037] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.205195 17589 tablet_bootstrap.cc:492] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:58.205835 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dd07adcbb1f4bd4bcb13bd089c89974" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:58.206025 16155 raft_consensus.cc:3037] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.206629 17589 tablet_bootstrap.cc:654] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.208923 16155 raft_consensus.cc:2454] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:58.209724 13383 leader_election.cc:302] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:58.210083 17519 raft_consensus.cc:2781] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:58.210265 17519 raft_consensus.cc:686] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:14:58.210412 16287 raft_consensus.cc:2454] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:14:58.210649 17519 consensus_queue.cc:237] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:14:58.213016 17576 leader_election.cc:288] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:14:58.214838 17585 tablet_bootstrap.cc:492] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:58.216440 17585 tablet_bootstrap.cc:654] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.219303 17589 tablet_bootstrap.cc:492] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:58.219560 17589 ts_tablet_manager.cc:1367] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20240502 14:14:58.220512 17589 raft_consensus.cc:348] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.220803 17589 raft_consensus.cc:374] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.220929 17589 raft_consensus.cc:729] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.221107 13040 catalog_manager.cc:5484] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:58.221720 17589 consensus_queue.cc:260] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.225684 17589 ts_tablet_manager.cc:1398] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 14:14:58.227205 17582 tablet_bootstrap.cc:492] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:58.227407 17582 ts_tablet_manager.cc:1367] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.003s
I20240502 14:14:58.229054 17582 raft_consensus.cc:348] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.229398 17582 raft_consensus.cc:374] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.229498 17582 raft_consensus.cc:729] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.229835 17582 consensus_queue.cc:260] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.231913 17582 ts_tablet_manager.cc:1398] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:14:58.248286 17229 raft_consensus.cc:1050] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER c86f7440570f4796babfd44c3d7fd847 to VOTER
I20240502 14:14:58.249249 17229 consensus_queue.cc:237] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:14:58.264218 17585 tablet_bootstrap.cc:492] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:58.264505 17585 ts_tablet_manager.cc:1367] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.012s
I20240502 14:14:58.265412 17585 raft_consensus.cc:348] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.265682 17585 raft_consensus.cc:374] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.265813 17585 raft_consensus.cc:729] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.266098 17585 consensus_queue.cc:260] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.266779 17585 ts_tablet_manager.cc:1398] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:58.276327 17570 raft_consensus.cc:482] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:58.276459 17570 raft_consensus.cc:504] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.277145 17570 leader_election.cc:288] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:58.277436 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e08376f3f74f709acda63bb2280575" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:58.277545 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e08376f3f74f709acda63bb2280575" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:58.277674 16155 raft_consensus.cc:2454] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:14:58.277786 16418 raft_consensus.cc:2454] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:14:58.278115 15956 leader_election.cc:302] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:58.278574 17570 raft_consensus.cc:2781] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:58.278724 17570 raft_consensus.cc:482] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:58.278908 17570 raft_consensus.cc:3037] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.282673 17570 raft_consensus.cc:504] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.282933 16155 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:58.283601 17399 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:58.283875 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e08376f3f74f709acda63bb2280575" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:58.284026 16155 raft_consensus.cc:3037] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.287264 16155 raft_consensus.cc:2454] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:14:58.287285 13315 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:58.287897 13446 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:58.288654 17398 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:58.288741 15956 leader_election.cc:302] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:58.289109 17563 raft_consensus.cc:2781] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:58.289304 17563 raft_consensus.cc:686] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:14:58.289294 17398 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:58.289644 17563 consensus_queue.cc:237] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.292506 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78e08376f3f74f709acda63bb2280575" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:14:58.292722 16418 raft_consensus.cc:3037] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.295507 16418 raft_consensus.cc:2454] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:14:58.296231 17570 leader_election.cc:288] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:58.309499 16418 consensus_queue.cc:237] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:14:58.330762 16155 raft_consensus.cc:1261] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:58.331952 17167 consensus_queue.cc:1035] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:14:58.332360 16355 consensus_peers.cc:485] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f961cf70d8ca482ba331241052833fd3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:58.339468 13580 raft_consensus.cc:1261] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:58.340132 17167 consensus_queue.cc:1035] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:58.343075 17167 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.348541 17606 raft_consensus.cc:2932] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:14:58.359623 17399 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:58.364799 13447 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:58.365897 13580 raft_consensus.cc:2932] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:14:58.368579 13040 catalog_manager.cc:5484] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:58.375726 13044 catalog_manager.cc:5484] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:58.385150 16155 raft_consensus.cc:2932] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:14:58.392279 13040 catalog_manager.cc:5484] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:58.401582 16152 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:58.421728 17606 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.429709 17700 ts_tablet_manager.cc:902] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:58.440471 13446 consensus_queue.cc:237] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:14:58.444762 13315 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:58.445577 13317 raft_consensus.cc:1261] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:14:58.446442 13382 consensus_peers.cc:485] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 536c22c444344bb485d3e522f0f77154. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:58.446653 17519 consensus_queue.cc:1035] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:58.449604 16287 raft_consensus.cc:1261] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:58.450326 17519 consensus_queue.cc:1035] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:58.470163 13580 consensus_queue.cc:237] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:14:58.472483 16154 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:58.473402 17685 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:58.473660 13447 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:58.477087 17229 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:58.479542 17128 consensus_queue.cc:1035] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.479542 17398 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:58.483418 17128 consensus_queue.cc:1035] T 4ea4a9b24178462dbe84e3c33f2d2245 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.484045 16154 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:58.484107 17519 raft_consensus.cc:2932] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:14:58.492259 13040 catalog_manager.cc:5484] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:58.493562 16284 raft_consensus.cc:2932] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:14:58.497999 13447 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:58.499599 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 652e49046212458280095c016a342f47 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:58.502642 13040 catalog_manager.cc:5484] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:14:58.503286 13315 raft_consensus.cc:2932] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:14:58.515501 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet 652e49046212458280095c016a342f47 with delete_type TABLET_DATA_TOMBSTONED (TS cd67493f6f0142f896010f2b8a39887d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:58.526288 17714 tablet_replica.cc:330] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:58.527561 17714 raft_consensus.cc:2229] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:58.528465 17714 raft_consensus.cc:2258] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:58.530961 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:58.533835 17716 ts_tablet_manager.cc:902] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:58.535163 17714 ts_tablet_manager.cc:1875] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:58.549345 13044 catalog_manager.cc:5576] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695) does not have the latest schema for tablet 78e08376f3f74f709acda63bb2280575 (table rebalancer_dynamic_table [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff]). Expected version 1 got 0
I20240502 14:14:58.554208 16420 raft_consensus.cc:1261] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:58.555651 17563 consensus_queue.cc:1035] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.572896 17714 ts_tablet_manager.cc:1888] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:14:58.577834 17714 log.cc:1196] T 652e49046212458280095c016a342f47 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/652e49046212458280095c016a342f47
I20240502 14:14:58.579103 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 652e49046212458280095c016a342f47 (table rebalance_test_table_11 [id=b7310fe1c01b4fe7b29328321c07e72d]) successfully deleted
I20240502 14:14:58.579726 17722 ts_tablet_manager.cc:902] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:58.583289 13044 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet 78e08376f3f74f709acda63bb2280575 (table rebalancer_dynamic_table [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff]). Expected version 1 got 0
I20240502 14:14:58.592993 17723 ts_tablet_manager.cc:902] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:58.593513 17731 tablet.cc:1722] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:58.594616 16154 raft_consensus.cc:1261] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:58.595467 13044 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet 78e08376f3f74f709acda63bb2280575 (table rebalancer_dynamic_table [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff]). Expected version 1 got 0
I20240502 14:14:58.595475 17570 consensus_queue.cc:1035] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.596994 17731 tablet.cc:1620] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:14:58.603394 17730 tablet.cc:1722] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:58.604079 17730 tablet.cc:1620] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:14:58.611799 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff] alter complete (version 1)
I20240502 14:14:58.622653 17734 tablet.cc:1722] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:58.625806 13044 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 78e08376f3f74f709acda63bb2280575 (table rebalancer_dynamic_table [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff]). Expected version 1 got 0
I20240502 14:14:58.639120 17734 tablet.cc:1620] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:58.648358 17727 ts_tablet_manager.cc:902] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:58.653447 17716 tablet_copy_client.cc:308] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:14:58.654788 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet 27857198d4ec430ebee897c90141c58d from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:53097)
I20240502 14:14:58.654971 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet 27857198d4ec430ebee897c90141c58d from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:53097: session id = 8b47d4390f8343a493d6c00123785968-27857198d4ec430ebee897c90141c58d
I20240502 14:14:58.657968 13601 tablet_copy_source_session.cc:205] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:58.659826 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27857198d4ec430ebee897c90141c58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.668303 17723 tablet_copy_client.cc:308] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:14:58.668684 17716 tablet_copy_client.cc:783] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:58.668933 17716 tablet_copy_client.cc:647] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:58.670845 16154 consensus_queue.cc:237] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:58.673215 17716 tablet_copy_client.cc:523] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:58.679219 17716 tablet_bootstrap.cc:492] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:58.682902 16420 raft_consensus.cc:1261] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:58.683738 13188 raft_consensus.cc:1261] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:58.684537 16804 consensus_queue.cc:1035] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:58.685192 16804 consensus_queue.cc:1035] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:14:58.689093 16804 raft_consensus.cc:2932] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:58.691404 17700 tablet_copy_client.cc:308] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:14:58.692706 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 0854e9f4851e400f9af18181a71ae8f7 from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:34141)
I20240502 14:14:58.692837 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 0854e9f4851e400f9af18181a71ae8f7 from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:34141: session id = 5297e816f782430d9a01220f0471a882-0854e9f4851e400f9af18181a71ae8f7
I20240502 14:14:58.693516 16420 raft_consensus.cc:2932] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:58.695665 13339 tablet_copy_source_session.cc:205] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:58.697031 17700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0854e9f4851e400f9af18181a71ae8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.697386 13188 raft_consensus.cc:2932] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:58.700191 13044 catalog_manager.cc:5484] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:14:58.704120 17700 tablet_copy_client.cc:783] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:58.704434 17700 tablet_copy_client.cc:647] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:58.707684 17700 tablet_copy_client.cc:523] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240502 14:14:58.711792 16090 consensus_peers.cc:485] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e2b8c72c7ba4be58f31ac073e41f2a8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:58.712769 17700 tablet_bootstrap.cc:492] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:58.716712 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet ab51b049c3674b629b55ed29df5b1fab from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:56181)
I20240502 14:14:58.716848 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet ab51b049c3674b629b55ed29df5b1fab from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:56181: session id = 5297e816f782430d9a01220f0471a882-ab51b049c3674b629b55ed29df5b1fab
I20240502 14:14:58.721225 13470 tablet_copy_source_session.cc:205] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:58.722935 17723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab51b049c3674b629b55ed29df5b1fab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.726712 17576 consensus_queue.cc:1035] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.730288 17723 tablet_copy_client.cc:783] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:58.730528 17723 tablet_copy_client.cc:647] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:58.731938 17723 tablet_copy_client.cc:523] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:58.733493 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:58.741662 17723 tablet_bootstrap.cc:492] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:58.743089 17730 tablet.cc:1722] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:58.743561 17730 tablet.cc:1620] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:14:58.744431 13044 catalog_manager.cc:5576] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695) does not have the latest schema for tablet 78e08376f3f74f709acda63bb2280575 (table rebalancer_dynamic_table [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff]). Expected version 2 got 1
I20240502 14:14:58.747924 17731 tablet.cc:1722] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:58.748492 17731 tablet.cc:1620] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:14:58.748889 17576 consensus_queue.cc:1035] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:58.753140 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:58.756415 17727 tablet_copy_client.cc:308] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:14:58.757238 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet f5bf499e8c044a1886989a90c6bb69fa from peer 88b77f6edc1c4a72b548bf47c05704f3 ({username='slave'} at 127.6.196.195:49767)
I20240502 14:14:58.757401 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet f5bf499e8c044a1886989a90c6bb69fa from peer 88b77f6edc1c4a72b548bf47c05704f3 at {username='slave'} at 127.6.196.195:49767: session id = 88b77f6edc1c4a72b548bf47c05704f3-f5bf499e8c044a1886989a90c6bb69fa
I20240502 14:14:58.760360 13046 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff] alter complete (version 2)
I20240502 14:14:58.760593 13601 tablet_copy_source_session.cc:205] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:58.764169 17727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5bf499e8c044a1886989a90c6bb69fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.765408 17734 tablet.cc:1722] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:58.765805 17734 tablet.cc:1620] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:58.775103 17727 tablet_copy_client.cc:783] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:58.775463 17727 tablet_copy_client.cc:647] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:58.777151 17727 tablet_copy_client.cc:523] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:58.780337 17727 tablet_bootstrap.cc:492] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:14:58.798931 17727 tablet_bootstrap.cc:492] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:58.799775 17727 tablet_bootstrap.cc:492] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap complete.
I20240502 14:14:58.800186 17727 ts_tablet_manager.cc:1367] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.004s
I20240502 14:14:58.801326 17727 raft_consensus.cc:348] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:14:58.801703 17727 raft_consensus.cc:729] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: LEARNER
I20240502 14:14:58.802023 17727 consensus_queue.cc:260] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:14:58.802803 17727 ts_tablet_manager.cc:1398] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:58.803640 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-f5bf499e8c044a1886989a90c6bb69fa received from {username='slave'} at 127.6.196.195:49767
I20240502 14:14:58.803778 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-f5bf499e8c044a1886989a90c6bb69fa on tablet f5bf499e8c044a1886989a90c6bb69fa with peer 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:14:58.818352 17722 tablet_copy_client.cc:308] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:14:58.819012 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet db06293ca0914bb88dbff3808e3f1874 from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:40701)
I20240502 14:14:58.819212 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet db06293ca0914bb88dbff3808e3f1874 from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:40701: session id = d1da7f11f0604f78934354a992bb0f55-db06293ca0914bb88dbff3808e3f1874
I20240502 14:14:58.829844 13601 tablet_copy_source_session.cc:205] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:58.837093 16265 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc251d4e82e452d989405e121d08dd4 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=bd0be61e86064cbca7fde36c32139e6a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.837261 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3da1778819497bb3bd56f9ba451046 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=bd0be61e86064cbca7fde36c32139e6a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.837663 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc251d4e82e452d989405e121d08dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.837715 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3da1778819497bb3bd56f9ba451046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.839094 16401 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc251d4e82e452d989405e121d08dd4 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=bd0be61e86064cbca7fde36c32139e6a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.839505 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc251d4e82e452d989405e121d08dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.841594 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc251d4e82e452d989405e121d08dd4 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=bd0be61e86064cbca7fde36c32139e6a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.841635 16267 tablet_service.cc:1467] Processing CreateTablet for tablet 5348ba5435c74f509ed77993191ef002 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=bd0be61e86064cbca7fde36c32139e6a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.841920 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc251d4e82e452d989405e121d08dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.842036 16267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5348ba5435c74f509ed77993191ef002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.845592 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 5348ba5435c74f509ed77993191ef002 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=bd0be61e86064cbca7fde36c32139e6a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.845705 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 5348ba5435c74f509ed77993191ef002 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=bd0be61e86064cbca7fde36c32139e6a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.846040 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5348ba5435c74f509ed77993191ef002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.846182 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5348ba5435c74f509ed77993191ef002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.846472 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3da1778819497bb3bd56f9ba451046 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=bd0be61e86064cbca7fde36c32139e6a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.846911 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3da1778819497bb3bd56f9ba451046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.849561 16400 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3da1778819497bb3bd56f9ba451046 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=bd0be61e86064cbca7fde36c32139e6a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:58.849890 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3da1778819497bb3bd56f9ba451046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.871872 17716 tablet_bootstrap.cc:492] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:58.872323 17716 tablet_bootstrap.cc:492] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:14:58.872660 17716 ts_tablet_manager.cc:1367] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.193s user 0.051s sys 0.002s
I20240502 14:14:58.885638 17722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db06293ca0914bb88dbff3808e3f1874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.886269 17768 tablet_bootstrap.cc:492] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:58.887305 17767 tablet_bootstrap.cc:492] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:58.888406 17768 tablet_bootstrap.cc:654] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.889514 17767 tablet_bootstrap.cc:654] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.885191 17769 tablet_bootstrap.cc:492] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:58.891685 17769 tablet_bootstrap.cc:654] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.893088 17771 tablet_bootstrap.cc:492] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:58.894114 17767 tablet_bootstrap.cc:492] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:58.894636 17767 ts_tablet_manager.cc:1367] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240502 14:14:58.894762 17771 tablet_bootstrap.cc:654] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.898309 17767 raft_consensus.cc:348] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:58.901338 17767 raft_consensus.cc:374] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.901692 17767 raft_consensus.cc:729] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.903280 17716 raft_consensus.cc:348] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:58.903524 17716 raft_consensus.cc:729] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:14:58.903865 17716 consensus_queue.cc:260] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:58.903312 17767 consensus_queue.cc:260] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:58.905699 17766 ts_tablet_manager.cc:902] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:14:58.906054 17716 ts_tablet_manager.cc:1398] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.033s user 0.004s sys 0.000s
I20240502 14:14:58.906520 17722 tablet_copy_client.cc:783] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:58.906783 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-27857198d4ec430ebee897c90141c58d received from {username='slave'} at 127.6.196.200:53097
I20240502 14:14:58.906894 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session 8b47d4390f8343a493d6c00123785968-27857198d4ec430ebee897c90141c58d on tablet 27857198d4ec430ebee897c90141c58d with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:14:58.904286 17765 tablet_bootstrap.cc:492] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:58.909581 17765 tablet_bootstrap.cc:654] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.909626 17770 ts_tablet_manager.cc:902] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d: Initiating tablet copy from peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:14:58.910020 17767 ts_tablet_manager.cc:1398] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 14:14:58.917495 17769 tablet_bootstrap.cc:492] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:58.917721 17769 ts_tablet_manager.cc:1367] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240502 14:14:58.918529 17768 tablet_bootstrap.cc:492] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:58.919368 17769 raft_consensus.cc:348] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:58.919700 17768 ts_tablet_manager.cc:1367] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240502 14:14:58.920689 17768 raft_consensus.cc:348] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:58.920991 17769 raft_consensus.cc:374] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.921144 17769 raft_consensus.cc:729] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.921519 17769 consensus_queue.cc:260] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:58.921917 17768 raft_consensus.cc:374] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.922848 17722 tablet_copy_client.cc:647] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:58.924403 17722 tablet_copy_client.cc:523] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:58.928858 17722 tablet_bootstrap.cc:492] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:58.934897 17765 tablet_bootstrap.cc:492] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:58.935089 17765 ts_tablet_manager.cc:1367] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20240502 14:14:58.935930 17765 raft_consensus.cc:348] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.935987 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:58.936278 17765 raft_consensus.cc:374] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.936493 17765 raft_consensus.cc:729] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.936774 17765 consensus_queue.cc:260] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.938446 17769 ts_tablet_manager.cc:1398] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240502 14:14:58.938612 17768 raft_consensus.cc:729] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.938868 17768 consensus_queue.cc:260] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:58.942865 17769 tablet_bootstrap.cc:492] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:58.944911 17768 ts_tablet_manager.cc:1398] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240502 14:14:58.946089 17730 tablet.cc:1722] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:58.946537 17771 tablet_bootstrap.cc:492] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:14:58.946713 17771 ts_tablet_manager.cc:1367] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.000s
I20240502 14:14:58.947741 17771 raft_consensus.cc:348] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:58.948064 17771 raft_consensus.cc:374] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.948227 17771 raft_consensus.cc:729] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.948542 17771 consensus_queue.cc:260] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:58.949331 17771 ts_tablet_manager.cc:1398] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:58.952086 17734 tablet.cc:1722] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:58.955682 17731 tablet.cc:1722] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:58.953521 17768 tablet_bootstrap.cc:492] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:58.958478 17768 tablet_bootstrap.cc:654] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.961937 17769 tablet_bootstrap.cc:654] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:58.968940 13045 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff] alter complete (version 3)
I20240502 14:14:58.986745 17700 tablet_bootstrap.cc:492] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:58.990658 17564 raft_consensus.cc:482] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:58.990799 17768 tablet_bootstrap.cc:492] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:14:58.990911 17769 tablet_bootstrap.cc:492] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:58.990970 17768 ts_tablet_manager.cc:1367] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.005s
I20240502 14:14:58.990877 17564 raft_consensus.cc:504] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:58.991091 17769 ts_tablet_manager.cc:1367] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.049s user 0.007s sys 0.000s
I20240502 14:14:58.991591 17564 leader_election.cc:288] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:58.991771 17768 raft_consensus.cc:348] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.991976 17768 raft_consensus.cc:374] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.992072 17768 raft_consensus.cc:729] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.991945 17769 raft_consensus.cc:348] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.992290 17769 raft_consensus.cc:374] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:58.992440 17769 raft_consensus.cc:729] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:58.992403 17768 consensus_queue.cc:260] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.992683 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3da1778819497bb3bd56f9ba451046" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:58.992908 13315 raft_consensus.cc:2454] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:14:58.993078 17768 ts_tablet_manager.cc:1398] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 14:14:58.993324 17770 tablet_copy_client.cc:308] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.200:43857
I20240502 14:14:58.993903 16442 tablet_copy_service.cc:140] P 8b47d4390f8343a493d6c00123785968: Received BeginTabletCopySession request for tablet f961cf70d8ca482ba331241052833fd3 from peer cd67493f6f0142f896010f2b8a39887d ({username='slave'} at 127.6.196.194:47545)
I20240502 14:14:58.994053 16442 tablet_copy_service.cc:161] P 8b47d4390f8343a493d6c00123785968: Beginning new tablet copy session on tablet f961cf70d8ca482ba331241052833fd3 from peer cd67493f6f0142f896010f2b8a39887d at {username='slave'} at 127.6.196.194:47545: session id = cd67493f6f0142f896010f2b8a39887d-f961cf70d8ca482ba331241052833fd3
I20240502 14:14:58.993973 17769 consensus_queue.cc:260] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:58.994131 15957 leader_election.cc:302] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:58.994464 17769 ts_tablet_manager.cc:1398] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:58.994459 17564 raft_consensus.cc:2781] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:58.994553 17564 raft_consensus.cc:482] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:58.994624 17564 raft_consensus.cc:3037] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:58.995321 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3da1778819497bb3bd56f9ba451046" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:58.996623 16442 tablet_copy_source_session.cc:205] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968: Tablet Copy: opened 0 blocks and 1 log segments
W20240502 14:14:58.998210 15956 leader_election.cc:341] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:58.998648 17770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f961cf70d8ca482ba331241052833fd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:58.999500 17700 tablet_bootstrap.cc:492] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882: Bootstrap complete.
I20240502 14:14:58.999778 17700 ts_tablet_manager.cc:1367] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.287s user 0.055s sys 0.007s
I20240502 14:14:58.997097 17564 raft_consensus.cc:504] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:59.002403 17564 leader_election.cc:288] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:59.002889 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3da1778819497bb3bd56f9ba451046" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:59.003062 13315 raft_consensus.cc:3037] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.002220 17700 raft_consensus.cc:348] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:59.005394 17700 raft_consensus.cc:729] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: LEARNER
I20240502 14:14:59.005594 17700 consensus_queue.cc:260] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:59.008796 17700 ts_tablet_manager.cc:1398] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 14:14:59.012735 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 5297e816f782430d9a01220f0471a882-0854e9f4851e400f9af18181a71ae8f7 received from {username='slave'} at 127.6.196.199:34141
I20240502 14:14:59.012876 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 5297e816f782430d9a01220f0471a882-0854e9f4851e400f9af18181a71ae8f7 on tablet 0854e9f4851e400f9af18181a71ae8f7 with peer 5297e816f782430d9a01220f0471a882
I20240502 14:14:59.014595 13315 raft_consensus.cc:2454] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:14:59.015008 15957 leader_election.cc:302] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:14:59.015357 17564 raft_consensus.cc:2781] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:59.015847 17564 raft_consensus.cc:686] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:14:59.016173 17564 consensus_queue.cc:237] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:59.017436 17765 ts_tablet_manager.cc:1398] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.082s user 0.001s sys 0.003s
I20240502 14:14:59.018144 16420 raft_consensus.cc:1203] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:59.018548 17765 tablet_bootstrap.cc:492] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:59.019332 17766 tablet_copy_client.cc:308] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:14:59.020005 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3da1778819497bb3bd56f9ba451046" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:14:59.020228 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet 536c22c444344bb485d3e522f0f77154 from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:43459)
I20240502 14:14:59.020412 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet 536c22c444344bb485d3e522f0f77154 from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:43459: session id = 20a967bad5504b1e9c6fb720ecdd1625-536c22c444344bb485d3e522f0f77154
W20240502 14:14:59.020895 15956 leader_election.cc:341] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:14:59.022692 17765 tablet_bootstrap.cc:654] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.023218 13470 tablet_copy_source_session.cc:205] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:59.026937 17766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 536c22c444344bb485d3e522f0f77154. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.029381 13042 catalog_manager.cc:5484] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:59.030396 17770 tablet_copy_client.cc:783] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:59.030711 17770 tablet_copy_client.cc:647] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:59.033136 17770 tablet_copy_client.cc:523] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:59.035158 17723 tablet_bootstrap.cc:492] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:59.035593 17723 tablet_bootstrap.cc:492] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882: Bootstrap complete.
I20240502 14:14:59.035856 17723 ts_tablet_manager.cc:1367] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.295s user 0.050s sys 0.014s
I20240502 14:14:59.036562 17723 raft_consensus.cc:348] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:59.036834 17723 raft_consensus.cc:729] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: LEARNER
I20240502 14:14:59.041697 17770 tablet_bootstrap.cc:492] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:59.042022 13580 consensus_queue.cc:237] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:59.042641 17723 consensus_queue.cc:260] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:59.043452 17723 ts_tablet_manager.cc:1398] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:14:59.043723 17766 tablet_copy_client.cc:783] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:59.044477 17766 tablet_copy_client.cc:647] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:59.046964 17766 tablet_copy_client.cc:523] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:59.049360 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session 5297e816f782430d9a01220f0471a882-ab51b049c3674b629b55ed29df5b1fab received from {username='slave'} at 127.6.196.199:56181
I20240502 14:14:59.049480 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session 5297e816f782430d9a01220f0471a882-ab51b049c3674b629b55ed29df5b1fab on tablet ab51b049c3674b629b55ed29df5b1fab with peer 5297e816f782430d9a01220f0471a882
I20240502 14:14:59.052613 17765 tablet_bootstrap.cc:492] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:59.053608 17765 ts_tablet_manager.cc:1367] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.000s
I20240502 14:14:59.054397 17766 tablet_bootstrap.cc:492] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:14:59.055212 17765 raft_consensus.cc:348] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:59.056655 17765 raft_consensus.cc:374] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.056921 17765 raft_consensus.cc:729] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.057353 17765 consensus_queue.cc:260] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
W20240502 14:14:59.059582 13515 consensus_peers.cc:485] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 766f50e1e5a44c8ba211a446bcf86705. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:59.060329 13188 raft_consensus.cc:1261] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:59.060736 13447 raft_consensus.cc:1261] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:59.060891 17794 consensus_queue.cc:1035] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:59.061396 17705 consensus_queue.cc:1035] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:59.067214 17770 tablet_bootstrap.cc:492] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:59.067727 17770 tablet_bootstrap.cc:492] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d: Bootstrap complete.
I20240502 14:14:59.068192 17770 ts_tablet_manager.cc:1367] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.006s
I20240502 14:14:59.069430 17765 ts_tablet_manager.cc:1398] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240502 14:14:59.069386 17770 raft_consensus.cc:348] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:14:59.069633 17770 raft_consensus.cc:729] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: LEARNER
I20240502 14:14:59.070171 17770 consensus_queue.cc:260] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:14:59.070762 17770 ts_tablet_manager.cc:1398] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:14:59.071399 13042 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:59.071583 13042 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:59.072613 17766 tablet_bootstrap.cc:492] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:59.072983 17766 tablet_bootstrap.cc:492] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:14:59.073158 17766 ts_tablet_manager.cc:1367] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.004s
I20240502 14:14:59.075234 13042 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 78e08376f3f74f709acda63bb2280575
I20240502 14:14:59.075870 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e08376f3f74f709acda63bb2280575 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:59 UTC) from {username='slave'} at 127.0.0.1:41222
I20240502 14:14:59.076660 17766 raft_consensus.cc:348] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:14:59.076833 17766 raft_consensus.cc:729] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:14:59.077775 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e08376f3f74f709acda63bb2280575 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:59 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:14:59.077786 17766 consensus_queue.cc:260] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:14:59.083079 17766 ts_tablet_manager.cc:1398] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240502 14:14:59.071398 16442 tablet_copy_service.cc:342] P 8b47d4390f8343a493d6c00123785968: Request end of tablet copy session cd67493f6f0142f896010f2b8a39887d-f961cf70d8ca482ba331241052833fd3 received from {username='slave'} at 127.6.196.194:47545
I20240502 14:14:59.084839 16442 tablet_copy_service.cc:433] P 8b47d4390f8343a493d6c00123785968: ending tablet copy session cd67493f6f0142f896010f2b8a39887d-f961cf70d8ca482ba331241052833fd3 on tablet f961cf70d8ca482ba331241052833fd3 with peer cd67493f6f0142f896010f2b8a39887d
I20240502 14:14:59.077966 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet 78e08376f3f74f709acda63bb2280575 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:59 UTC) from {username='slave'} at 127.0.0.1:53312
I20240502 14:14:59.087277 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-536c22c444344bb485d3e522f0f77154 received from {username='slave'} at 127.6.196.193:43459
I20240502 14:14:59.087419 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-536c22c444344bb485d3e522f0f77154 on tablet 536c22c444344bb485d3e522f0f77154 with peer 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:14:59.093626 17805 tablet_replica.cc:330] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:14:59.093845 17804 tablet_replica.cc:330] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:14:59.094317 17804 raft_consensus.cc:2229] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:59.095048 17804 raft_consensus.cc:2258] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:59.095048 17805 raft_consensus.cc:2229] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:59.095311 17805 raft_consensus.cc:2258] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:59.096649 17805 ts_tablet_manager.cc:1875] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:59.097453 17804 ts_tablet_manager.cc:1875] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:59.070365 17765 tablet_bootstrap.cc:492] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:59.101465 17765 tablet_bootstrap.cc:654] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.105197 17804 ts_tablet_manager.cc:1888] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:59.105448 17804 log.cc:1196] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/78e08376f3f74f709acda63bb2280575
I20240502 14:14:59.106134 17804 ts_tablet_manager.cc:1909] T 78e08376f3f74f709acda63bb2280575 P d1da7f11f0604f78934354a992bb0f55: Deleting consensus metadata
I20240502 14:14:59.105844 17806 tablet_replica.cc:330] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:14:59.107564 17806 raft_consensus.cc:2229] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:59.107929 17806 raft_consensus.cc:2258] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:59.108876 17806 ts_tablet_manager.cc:1875] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:59.109500 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet 78e08376f3f74f709acda63bb2280575 (table rebalancer_dynamic_table_new_name [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff]) successfully deleted
I20240502 14:14:59.112061 17805 ts_tablet_manager.cc:1888] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:59.112181 17805 log.cc:1196] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/78e08376f3f74f709acda63bb2280575
I20240502 14:14:59.112629 17805 ts_tablet_manager.cc:1909] T 78e08376f3f74f709acda63bb2280575 P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:14:59.114197 17794 raft_consensus.cc:2932] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:59.118266 13446 raft_consensus.cc:2932] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:59.124348 13044 catalog_manager.cc:5484] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:59.126964 17722 tablet_bootstrap.cc:492] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:59.127362 17722 tablet_bootstrap.cc:492] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:14:59.128968 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 78e08376f3f74f709acda63bb2280575 (table rebalancer_dynamic_table_new_name [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff]) successfully deleted
I20240502 14:14:59.131004 17722 ts_tablet_manager.cc:1367] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.202s user 0.045s sys 0.024s
I20240502 14:14:59.131774 17722 raft_consensus.cc:348] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:59.132056 17722 raft_consensus.cc:729] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:14:59.132241 17722 consensus_queue.cc:260] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:59.137159 17722 ts_tablet_manager.cc:1398] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:14:59.140455 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-db06293ca0914bb88dbff3808e3f1874 received from {username='slave'} at 127.6.196.197:40701
I20240502 14:14:59.140575 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-db06293ca0914bb88dbff3808e3f1874 on tablet db06293ca0914bb88dbff3808e3f1874 with peer d1da7f11f0604f78934354a992bb0f55
I20240502 14:14:59.142635 17765 tablet_bootstrap.cc:492] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:59.142812 17765 ts_tablet_manager.cc:1367] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.000s
I20240502 14:14:59.143491 17506 raft_consensus.cc:482] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:59.143651 17506 raft_consensus.cc:504] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:59.143786 17765 raft_consensus.cc:348] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:59.144047 17765 raft_consensus.cc:374] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.144271 17765 raft_consensus.cc:729] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.144315 17506 leader_election.cc:288] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:59.144623 17765 consensus_queue.cc:260] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:59.145231 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5348ba5435c74f509ed77993191ef002" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:59.145193 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5348ba5435c74f509ed77993191ef002" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:59.145306 17765 ts_tablet_manager.cc:1398] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240502 14:14:59.145431 16420 raft_consensus.cc:2454] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:14:59.145437 16152 raft_consensus.cc:2454] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:14:59.145795 16220 leader_election.cc:302] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:59.146311 13447 raft_consensus.cc:1203] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:59.146420 17506 raft_consensus.cc:2781] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:59.146528 17506 raft_consensus.cc:482] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:59.146600 17506 raft_consensus.cc:3037] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.148706 17506 raft_consensus.cc:504] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:59.150027 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5348ba5435c74f509ed77993191ef002" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:14:59.150254 16420 raft_consensus.cc:3037] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.154366 16420 raft_consensus.cc:2454] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:14:59.155247 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5348ba5435c74f509ed77993191ef002" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:59.155512 16152 raft_consensus.cc:3037] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.159978 16152 raft_consensus.cc:2454] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:14:59.160497 16220 leader_election.cc:302] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:59.161417 17506 leader_election.cc:288] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:59.161911 17506 raft_consensus.cc:2781] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:59.162283 17506 raft_consensus.cc:686] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:14:59.163669 17806 ts_tablet_manager.cc:1888] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:59.163487 17506 consensus_queue.cc:237] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:14:59.163863 17806 log.cc:1196] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/78e08376f3f74f709acda63bb2280575
I20240502 14:14:59.164309 17806 ts_tablet_manager.cc:1909] T 78e08376f3f74f709acda63bb2280575 P 8b47d4390f8343a493d6c00123785968: Deleting consensus metadata
I20240502 14:14:59.165272 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 78e08376f3f74f709acda63bb2280575 (table rebalancer_dynamic_table_new_name [id=8dc1ea2ab6ee49109d4dbc3333b3f0ff]) successfully deleted
I20240502 14:14:59.176457 13044 catalog_manager.cc:5484] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:59.178449 13045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:59.182279 13186 raft_consensus.cc:2932] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:59.191558 13580 consensus_queue.cc:237] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:59.192718 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 725c627ceb4944c8b9f05f4c12d28696 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=70bc6c11b95a4cfabbd48fc2c3ce04f2]), partition=
I20240502 14:14:59.193094 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725c627ceb4944c8b9f05f4c12d28696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.193454 16401 tablet_service.cc:1467] Processing CreateTablet for tablet 725c627ceb4944c8b9f05f4c12d28696 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=70bc6c11b95a4cfabbd48fc2c3ce04f2]), partition=
I20240502 14:14:59.193814 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725c627ceb4944c8b9f05f4c12d28696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.195876 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 725c627ceb4944c8b9f05f4c12d28696 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=70bc6c11b95a4cfabbd48fc2c3ce04f2]), partition=
I20240502 14:14:59.196228 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 725c627ceb4944c8b9f05f4c12d28696. 1 dirs total, 0 dirs full, 0 dirs failed
W20240502 14:14:59.199391 13516 consensus_peers.cc:485] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 80040c60b029481285c50171bcc5ce35. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:59.200347 13447 raft_consensus.cc:1261] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:59.200924 13315 raft_consensus.cc:1261] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:59.200870 17705 consensus_queue.cc:1035] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:59.201429 17705 consensus_queue.cc:1035] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:59.202031 17765 tablet_bootstrap.cc:492] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:14:59.203474 17765 tablet_bootstrap.cc:654] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.205020 17769 tablet_bootstrap.cc:492] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:59.206692 17767 tablet_bootstrap.cc:492] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:14:59.206975 17769 tablet_bootstrap.cc:654] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.208597 17767 tablet_bootstrap.cc:654] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:14:59.219360 17765 tablet_bootstrap.cc:492] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:14:59.219617 17765 ts_tablet_manager.cc:1367] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20240502 14:14:59.219720 17769 tablet_bootstrap.cc:492] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:14:59.219899 17769 ts_tablet_manager.cc:1367] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.005s
I20240502 14:14:59.220587 17765 raft_consensus.cc:348] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:59.220973 17765 raft_consensus.cc:374] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.221097 17765 raft_consensus.cc:729] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.221486 17765 consensus_queue.cc:260] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:59.220799 17769 raft_consensus.cc:348] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:59.221978 17769 raft_consensus.cc:374] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.222122 17769 raft_consensus.cc:729] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.222368 17769 consensus_queue.cc:260] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:59.223062 17765 ts_tablet_manager.cc:1398] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:14:59.229782 17767 tablet_bootstrap.cc:492] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:14:59.230075 17767 ts_tablet_manager.cc:1367] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.008s
I20240502 14:14:59.231600 17767 raft_consensus.cc:348] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:59.231985 17767 raft_consensus.cc:374] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:14:59.232110 17767 raft_consensus.cc:729] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:14:59.232399 17767 consensus_queue.cc:260] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:59.232627 17769 ts_tablet_manager.cc:1398] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240502 14:14:59.237607 17767 ts_tablet_manager.cc:1398] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:14:59.245832 17705 raft_consensus.cc:2932] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:59.246325 13447 raft_consensus.cc:2932] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:59.251030 13317 raft_consensus.cc:2932] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:59.256312 13045 catalog_manager.cc:5484] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:14:59.274763 17722 ts_tablet_manager.cc:902] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:59.275357 16288 raft_consensus.cc:1203] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:59.280174 17720 raft_consensus.cc:482] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:59.280336 17720 raft_consensus.cc:504] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:59.280951 17720 leader_election.cc:288] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:59.281607 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725c627ceb4944c8b9f05f4c12d28696" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:14:59.281919 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725c627ceb4944c8b9f05f4c12d28696" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:14:59.282138 16420 raft_consensus.cc:2454] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:14:59.282492 13317 raft_consensus.cc:2454] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:14:59.282680 16089 leader_election.cc:302] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:59.283149 17720 raft_consensus.cc:2781] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:59.283356 17720 raft_consensus.cc:482] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:59.283466 17720 raft_consensus.cc:3037] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.286051 17720 raft_consensus.cc:504] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:59.286862 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725c627ceb4944c8b9f05f4c12d28696" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:14:59.287052 16420 raft_consensus.cc:3037] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.291695 17720 leader_election.cc:288] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:14:59.292203 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "725c627ceb4944c8b9f05f4c12d28696" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:14:59.292451 13317 raft_consensus.cc:3037] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.292552 16420 raft_consensus.cc:2454] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:14:59.292879 16089 leader_election.cc:302] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:14:59.293138 17720 raft_consensus.cc:2781] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:59.293292 17720 raft_consensus.cc:686] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:14:59.293541 17720 consensus_queue.cc:237] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:14:59.295722 13317 raft_consensus.cc:2454] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:14:59.305341 13045 catalog_manager.cc:5484] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:14:59.321537 17722 tablet_copy_client.cc:308] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.198:42041
I20240502 14:14:59.322645 16175 tablet_copy_service.cc:140] P c86f7440570f4796babfd44c3d7fd847: Received BeginTabletCopySession request for tablet 1e2b8c72c7ba4be58f31ac073e41f2a8 from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:52807)
I20240502 14:14:59.322892 16175 tablet_copy_service.cc:161] P c86f7440570f4796babfd44c3d7fd847: Beginning new tablet copy session on tablet 1e2b8c72c7ba4be58f31ac073e41f2a8 from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:52807: session id = d1da7f11f0604f78934354a992bb0f55-1e2b8c72c7ba4be58f31ac073e41f2a8
I20240502 14:14:59.329528 16175 tablet_copy_source_session.cc:205] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:59.331089 17722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e2b8c72c7ba4be58f31ac073e41f2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.335292 17780 raft_consensus.cc:482] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:14:59.335451 17780 raft_consensus.cc:504] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:59.336122 17780 leader_election.cc:288] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:59.336863 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc251d4e82e452d989405e121d08dd4" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:14:59.337059 16288 raft_consensus.cc:2454] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:14:59.337721 17722 tablet_copy_client.cc:783] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:59.337957 17722 tablet_copy_client.cc:647] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:59.339526 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc251d4e82e452d989405e121d08dd4" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:14:59.339732 16152 raft_consensus.cc:2454] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:14:59.341579 16355 leader_election.cc:302] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:14:59.342952 17722 tablet_copy_client.cc:523] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:59.343062 17780 raft_consensus.cc:2781] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:14:59.343175 17780 raft_consensus.cc:482] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:14:59.343252 17780 raft_consensus.cc:3037] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.345762 17780 raft_consensus.cc:504] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:59.346086 17722 tablet_bootstrap.cc:492] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:14:59.346771 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc251d4e82e452d989405e121d08dd4" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:14:59.346974 16288 raft_consensus.cc:3037] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.348270 17780 leader_election.cc:288] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:14:59.349706 16288 raft_consensus.cc:2454] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:14:59.350622 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fc251d4e82e452d989405e121d08dd4" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:14:59.350941 16152 raft_consensus.cc:3037] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.353701 16355 leader_election.cc:302] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:14:59.355321 17780 raft_consensus.cc:2781] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:14:59.355561 17780 raft_consensus.cc:686] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:14:59.355883 17780 consensus_queue.cc:237] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:14:59.355089 16152 raft_consensus.cc:2454] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:14:59.358639 17722 tablet_bootstrap.cc:492] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:59.359061 17722 tablet_bootstrap.cc:492] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:14:59.359380 17722 ts_tablet_manager.cc:1367] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.005s
I20240502 14:14:59.359695 13045 catalog_manager.cc:5484] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } }
I20240502 14:14:59.360152 17722 raft_consensus.cc:348] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:59.360503 17722 raft_consensus.cc:729] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:14:59.360706 17722 consensus_queue.cc:260] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:14:59.362015 17722 ts_tablet_manager.cc:1398] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 14:14:59.362618 16175 tablet_copy_service.cc:342] P c86f7440570f4796babfd44c3d7fd847: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-1e2b8c72c7ba4be58f31ac073e41f2a8 received from {username='slave'} at 127.6.196.197:52807
I20240502 14:14:59.362723 16175 tablet_copy_service.cc:433] P c86f7440570f4796babfd44c3d7fd847: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-1e2b8c72c7ba4be58f31ac073e41f2a8 on tablet 1e2b8c72c7ba4be58f31ac073e41f2a8 with peer d1da7f11f0604f78934354a992bb0f55
I20240502 14:14:59.398085 13317 raft_consensus.cc:1203] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:59.411813 13186 raft_consensus.cc:1203] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:59.412508 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:14:59.419763 13317 raft_consensus.cc:1261] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:59.420837 17720 consensus_queue.cc:1035] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.421448 16420 raft_consensus.cc:1261] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:14:59.424731 17720 consensus_queue.cc:1035] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.438434 16288 raft_consensus.cc:1203] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:59.452936 17734 tablet.cc:1722] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:59.453156 13045 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet 725c627ceb4944c8b9f05f4c12d28696 (table rebalancer_dynamic_table [id=70bc6c11b95a4cfabbd48fc2c3ce04f2]). Expected version 1 got 0
I20240502 14:14:59.453498 17734 tablet.cc:1620] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:59.453670 13042 catalog_manager.cc:5576] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193) does not have the latest schema for tablet 725c627ceb4944c8b9f05f4c12d28696 (table rebalancer_dynamic_table [id=70bc6c11b95a4cfabbd48fc2c3ce04f2]). Expected version 1 got 0
I20240502 14:14:59.456352 17869 tablet.cc:1722] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:59.456920 17869 tablet.cc:1620] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: MemRowSet was empty: no flush needed.
I20240502 14:14:59.458844 13045 catalog_manager.cc:5576] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193) does not have the latest schema for tablet 725c627ceb4944c8b9f05f4c12d28696 (table rebalancer_dynamic_table [id=70bc6c11b95a4cfabbd48fc2c3ce04f2]). Expected version 1 got 0
I20240502 14:14:59.461546 17731 tablet.cc:1722] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:14:59.462033 17731 tablet.cc:1620] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:14:59.462253 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=70bc6c11b95a4cfabbd48fc2c3ce04f2] alter complete (version 1)
I20240502 14:14:59.464531 13042 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet 725c627ceb4944c8b9f05f4c12d28696 (table rebalancer_dynamic_table [id=70bc6c11b95a4cfabbd48fc2c3ce04f2]). Expected version 1 got 0
I20240502 14:14:59.488766 16021 raft_consensus.cc:1203] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:14:59.496490 16420 raft_consensus.cc:3037] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:14:59.499505 17564 consensus_queue.cc:1035] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.505383 17570 consensus_queue.cc:1035] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.554137 13580 consensus_queue.cc:237] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:59.556108 13187 raft_consensus.cc:1261] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:59.556411 13447 raft_consensus.cc:1261] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:14:59.556461 13515 consensus_peers.cc:485] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9ad2eb265e0f469787eb1699e339e0ed. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:59.556630 17705 consensus_queue.cc:1035] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:59.556926 17398 consensus_queue.cc:1035] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:59.559284 17685 raft_consensus.cc:2932] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:59.559913 13187 raft_consensus.cc:2932] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:59.559914 13447 raft_consensus.cc:2932] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:59.565393 13045 catalog_manager.cc:5484] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:14:59.570531 17506 consensus_queue.cc:1035] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.572376 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:14:59.581449 17734 tablet.cc:1722] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:59.581996 17734 tablet.cc:1620] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:14:59.585816 13317 consensus_queue.cc:237] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:14:59.586364 17731 tablet.cc:1722] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:59.586776 17731 tablet.cc:1620] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:14:59.585950 17869 tablet.cc:1722] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:14:59.587458 17869 tablet.cc:1620] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: MemRowSet was empty: no flush needed.
I20240502 14:14:59.594151 13447 raft_consensus.cc:1261] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:59.594795 13579 raft_consensus.cc:1261] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:59.594792 17738 consensus_queue.cc:1035] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:14:59.596385 13252 consensus_peers.cc:485] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 21ac2a9fa2154a0bbbe15aeaf178ea1a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:14:59.596284 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=70bc6c11b95a4cfabbd48fc2c3ce04f2] alter complete (version 2)
I20240502 14:14:59.596755 17813 consensus_queue.cc:1035] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.596969 17927 ts_tablet_manager.cc:902] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:59.598891 17738 consensus_queue.cc:1035] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:59.604627 17685 raft_consensus.cc:1050] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 to VOTER
I20240502 14:14:59.605607 17685 consensus_queue.cc:237] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:14:59.620615 16021 raft_consensus.cc:1261] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:59.621418 17398 consensus_queue.cc:1035] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:59.629428 13446 raft_consensus.cc:1261] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:59.630506 17705 consensus_queue.cc:1035] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:59.630414 17738 raft_consensus.cc:2932] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:59.631274 17927 tablet_copy_client.cc:308] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:14:59.631953 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet 766f50e1e5a44c8ba211a446bcf86705 from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:42449)
I20240502 14:14:59.632091 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet 766f50e1e5a44c8ba211a446bcf86705 from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:42449: session id = c86f7440570f4796babfd44c3d7fd847-766f50e1e5a44c8ba211a446bcf86705
I20240502 14:14:59.632944 13446 raft_consensus.cc:2932] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:59.633430 17794 raft_consensus.cc:1050] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:14:59.634627 17794 consensus_queue.cc:237] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:14:59.635031 13601 tablet_copy_source_session.cc:205] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:59.638751 17927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 766f50e1e5a44c8ba211a446bcf86705. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.639271 16418 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:59.639494 13445 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:59.640522 13315 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:59.640910 17705 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:59.641593 17705 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:59.641796 13044 catalog_manager.cc:5484] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:59.642184 17705 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:59.650985 17927 tablet_copy_client.cc:783] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:59.651227 17927 tablet_copy_client.cc:647] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:59.652681 17927 tablet_copy_client.cc:523] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:59.656142 17927 tablet_bootstrap.cc:492] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:14:59.662741 16022 raft_consensus.cc:1203] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:14:59.664467 13579 raft_consensus.cc:2932] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:14:59.664979 17398 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:59.670872 16418 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:59.677958 13042 catalog_manager.cc:5484] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:59.679903 17946 ts_tablet_manager.cc:902] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:14:59.680359 16420 raft_consensus.cc:1261] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:59.681762 13445 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:59.689664 17398 consensus_queue.cc:1035] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:59.690608 17945 raft_consensus.cc:2932] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:59.691027 13579 consensus_queue.cc:237] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:14:59.694614 16022 raft_consensus.cc:2932] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:59.699450 16419 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:59.700719 17685 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:59.702507 13447 raft_consensus.cc:2932] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:59.703872 13042 catalog_manager.cc:5484] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:14:59.707324 13579 consensus_queue.cc:237] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:14:59.709630 16021 raft_consensus.cc:1261] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:59.710121 17705 consensus_queue.cc:1035] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:59.710587 13447 raft_consensus.cc:1261] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:14:59.711040 16420 raft_consensus.cc:2932] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:59.711898 17705 consensus_queue.cc:1035] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:14:59.711804 17685 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:59.713322 17705 raft_consensus.cc:2932] T f5bf499e8c044a1886989a90c6bb69fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:59.714510 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:14:59.713935 16419 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:59.718051 13447 raft_consensus.cc:2932] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:59.721146 13449 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:59.721913 17927 tablet_bootstrap.cc:492] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:59.722386 17927 tablet_bootstrap.cc:492] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:14:59.722628 17927 ts_tablet_manager.cc:1367] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.067s user 0.036s sys 0.010s
I20240502 14:14:59.723276 17398 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:59.723659 17927 raft_consensus.cc:348] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:59.723892 17927 raft_consensus.cc:729] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:14:59.724092 17927 consensus_queue.cc:260] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:14:59.724604 17927 ts_tablet_manager.cc:1398] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:14:59.725296 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-766f50e1e5a44c8ba211a446bcf86705 received from {username='slave'} at 127.6.196.198:42449
I20240502 14:14:59.725421 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-766f50e1e5a44c8ba211a446bcf86705 on tablet 766f50e1e5a44c8ba211a446bcf86705 with peer c86f7440570f4796babfd44c3d7fd847
I20240502 14:14:59.725919 16021 raft_consensus.cc:2932] T f5bf499e8c044a1886989a90c6bb69fa P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:59.728542 17734 tablet.cc:1722] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:59.731278 13445 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:59.733846 13044 catalog_manager.cc:5484] T f5bf499e8c044a1886989a90c6bb69fa P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:14:59.734716 13042 catalog_manager.cc:5484] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:14:59.739688 13045 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=70bc6c11b95a4cfabbd48fc2c3ce04f2] alter complete (version 3)
I20240502 14:14:59.740463 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet 27857198d4ec430ebee897c90141c58d with delete_type TABLET_DATA_TOMBSTONED (TS cd67493f6f0142f896010f2b8a39887d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:59.741194 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet f5bf499e8c044a1886989a90c6bb69fa with delete_type TABLET_DATA_TOMBSTONED (TS 8b47d4390f8343a493d6c00123785968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53312
I20240502 14:14:59.745352 17869 tablet.cc:1722] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:59.745654 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27857198d4ec430ebee897c90141c58d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:59.746062 17731 tablet.cc:1722] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:14:59.746799 17953 tablet_replica.cc:330] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:59.747030 17953 raft_consensus.cc:2229] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:59.747196 17953 pending_rounds.cc:62] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d: Trying to abort 1 pending ops.
I20240502 14:14:59.747293 17953 pending_rounds.cc:68] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023244491300052992 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "27857198d4ec430ebee897c90141c58d" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } } }
I20240502 14:14:59.747674 17953 raft_consensus.cc:2894] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 14:14:59.747799 17953 raft_consensus.cc:2866] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:14:59.747910 17953 raft_consensus.cc:2258] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:59.751058 17953 ts_tablet_manager.cc:1875] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:59.752249 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f5bf499e8c044a1886989a90c6bb69fa with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:59.759779 17954 tablet_replica.cc:330] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:14:59.760140 17954 raft_consensus.cc:2229] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:59.760444 17954 raft_consensus.cc:2258] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:59.761391 17954 ts_tablet_manager.cc:1875] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:59.761667 17946 tablet_copy_client.cc:308] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:14:59.762598 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet 80040c60b029481285c50171bcc5ce35 from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:53105)
I20240502 14:14:59.762761 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet 80040c60b029481285c50171bcc5ce35 from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:53105: session id = 5297e816f782430d9a01220f0471a882-80040c60b029481285c50171bcc5ce35
I20240502 14:14:59.765004 17953 ts_tablet_manager.cc:1888] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:14:59.765117 17953 log.cc:1196] T 27857198d4ec430ebee897c90141c58d P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/27857198d4ec430ebee897c90141c58d
I20240502 14:14:59.765280 13601 tablet_copy_source_session.cc:205] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:14:59.767767 17954 ts_tablet_manager.cc:1888] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:14:59.767971 17954 log.cc:1196] T f5bf499e8c044a1886989a90c6bb69fa P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/f5bf499e8c044a1886989a90c6bb69fa
I20240502 14:14:59.768784 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet f5bf499e8c044a1886989a90c6bb69fa (table rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]) successfully deleted
I20240502 14:14:59.770568 17946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80040c60b029481285c50171bcc5ce35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.771214 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 27857198d4ec430ebee897c90141c58d (table rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]) successfully deleted
I20240502 14:14:59.775835 17946 tablet_copy_client.cc:783] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:14:59.776053 17946 tablet_copy_client.cc:647] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:14:59.777336 17946 tablet_copy_client.cc:523] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:14:59.779476 17946 tablet_bootstrap.cc:492] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:14:59.815295 17514 raft_consensus.cc:1050] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 5297e816f782430d9a01220f0471a882 to VOTER
I20240502 14:14:59.815956 17514 consensus_queue.cc:237] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:14:59.819267 13445 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:59.819298 16288 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:59.819779 13579 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:59.820034 17940 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:59.820343 17738 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:59.820576 17946 tablet_bootstrap.cc:492] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:14:59.820844 17940 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:59.821116 17946 tablet_bootstrap.cc:492] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882: Bootstrap complete.
I20240502 14:14:59.821390 17946 ts_tablet_manager.cc:1367] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.042s user 0.039s sys 0.002s
I20240502 14:14:59.822177 17946 raft_consensus.cc:348] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:59.822386 17946 raft_consensus.cc:729] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: LEARNER
I20240502 14:14:59.822571 17946 consensus_queue.cc:260] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:14:59.823179 17946 ts_tablet_manager.cc:1398] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:14:59.824697 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session 5297e816f782430d9a01220f0471a882-80040c60b029481285c50171bcc5ce35 received from {username='slave'} at 127.6.196.199:53105
I20240502 14:14:59.824807 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session 5297e816f782430d9a01220f0471a882-80040c60b029481285c50171bcc5ce35 on tablet 80040c60b029481285c50171bcc5ce35 with peer 5297e816f782430d9a01220f0471a882
I20240502 14:14:59.829382 17940 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:59.831100 16288 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:59.833101 17786 consensus_queue.cc:1035] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.838320 13044 catalog_manager.cc:5484] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:59.843230 13036 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:59.843458 13036 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:14:59.846527 17780 consensus_queue.cc:1035] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:14:59.854981 13579 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:59.855271 13445 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:59.861265 13036 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 725c627ceb4944c8b9f05f4c12d28696
I20240502 14:14:59.864148 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet 725c627ceb4944c8b9f05f4c12d28696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:59 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:14:59.864192 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet 725c627ceb4944c8b9f05f4c12d28696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:59 UTC) from {username='slave'} at 127.0.0.1:53312
I20240502 14:14:59.864879 17954 tablet_replica.cc:330] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:14:59.865154 17954 raft_consensus.cc:2229] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:59.865417 17954 raft_consensus.cc:2258] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:59.866605 17954 ts_tablet_manager.cc:1875] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:59.871917 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet 725c627ceb4944c8b9f05f4c12d28696 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:14:59 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:14:59.872187 17953 tablet_replica.cc:330] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:14:59.872357 17953 raft_consensus.cc:2229] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:59.872553 17953 raft_consensus.cc:2258] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:59.873420 17953 ts_tablet_manager.cc:1875] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:59.878434 13315 consensus_queue.cc:237] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:14:59.883935 17954 ts_tablet_manager.cc:1888] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:59.884182 17954 log.cc:1196] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/725c627ceb4944c8b9f05f4c12d28696
I20240502 14:14:59.883095 16288 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:59.884843 17954 ts_tablet_manager.cc:1909] T 725c627ceb4944c8b9f05f4c12d28696 P 8b47d4390f8343a493d6c00123785968: Deleting consensus metadata
I20240502 14:14:59.884984 17940 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:59.886890 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 725c627ceb4944c8b9f05f4c12d28696 (table rebalancer_dynamic_table_new_name [id=70bc6c11b95a4cfabbd48fc2c3ce04f2]) successfully deleted
I20240502 14:14:59.886767 13579 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:14:59.887912 17514 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:59.889681 17966 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:14:59.891088 17953 ts_tablet_manager.cc:1888] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:59.891394 13579 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:59.893560 17519 raft_consensus.cc:1050] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER 5297e816f782430d9a01220f0471a882 to VOTER
I20240502 14:14:59.894228 17519 consensus_queue.cc:237] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:14:59.895313 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0854e9f4851e400f9af18181a71ae8f7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:14:59.896920 17974 tablet_replica.cc:330] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:14:59.897173 17974 raft_consensus.cc:2229] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:14:59.896992 13045 catalog_manager.cc:5484] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:59.897524 17974 raft_consensus.cc:2258] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:59.899628 17953 log.cc:1196] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/725c627ceb4944c8b9f05f4c12d28696
I20240502 14:14:59.900722 17953 ts_tablet_manager.cc:1909] T 725c627ceb4944c8b9f05f4c12d28696 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:14:59.901842 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 0854e9f4851e400f9af18181a71ae8f7 with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42656
I20240502 14:14:59.902526 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 725c627ceb4944c8b9f05f4c12d28696 (table rebalancer_dynamic_table_new_name [id=70bc6c11b95a4cfabbd48fc2c3ce04f2]) successfully deleted
I20240502 14:14:59.892854 16288 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:59.904973 17974 ts_tablet_manager.cc:1875] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:14:59.909659 13042 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 0854e9f4851e400f9af18181a71ae8f7 with cas_config_opid_index 5: aborting the task: latest config opid_index 6; task opid_index 5
I20240502 14:14:59.911346 13187 raft_consensus.cc:1261] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:59.912178 17519 consensus_queue.cc:1035] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:59.912402 16285 raft_consensus.cc:1261] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:59.913503 13579 raft_consensus.cc:1261] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:14:59.913787 17086 consensus_queue.cc:1035] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:59.914161 17576 consensus_queue.cc:1035] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:14:59.918102 17976 tablet_replica.cc:330] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:14:59.918491 17976 raft_consensus.cc:2229] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:14:59.919454 17976 raft_consensus.cc:2258] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:14:59.919791 17974 ts_tablet_manager.cc:1888] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:14:59.920032 17974 log.cc:1196] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/725c627ceb4944c8b9f05f4c12d28696
I20240502 14:14:59.920766 17974 ts_tablet_manager.cc:1909] T 725c627ceb4944c8b9f05f4c12d28696 P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:14:59.921610 17976 ts_tablet_manager.cc:1875] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:14:59.921969 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 725c627ceb4944c8b9f05f4c12d28696 (table rebalancer_dynamic_table_new_name [id=70bc6c11b95a4cfabbd48fc2c3ce04f2]) successfully deleted
I20240502 14:14:59.934197 17976 ts_tablet_manager.cc:1888] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:14:59.934856 16285 raft_consensus.cc:2932] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:59.937517 17780 raft_consensus.cc:1050] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968: attempting to promote NON_VOTER cd67493f6f0142f896010f2b8a39887d to VOTER
I20240502 14:14:59.938416 17780 consensus_queue.cc:237] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:14:59.939349 17976 log.cc:1196] T 0854e9f4851e400f9af18181a71ae8f7 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/0854e9f4851e400f9af18181a71ae8f7
I20240502 14:14:59.939406 13042 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:14:59.940187 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 0854e9f4851e400f9af18181a71ae8f7 (table rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]) successfully deleted
I20240502 14:14:59.933916 17086 raft_consensus.cc:2932] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:59.946041 16152 raft_consensus.cc:1261] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:59.948787 17983 consensus_queue.cc:1035] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:59.949648 13580 raft_consensus.cc:1261] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:59.951467 17790 consensus_queue.cc:1035] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:59.953940 16267 tablet_service.cc:1467] Processing CreateTablet for tablet 748564520c0c4ebda80e6f8c7f9acd13 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9dc6b07ff15645ab9222cc55c5a4a00b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:59.954154 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 748564520c0c4ebda80e6f8c7f9acd13 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9dc6b07ff15645ab9222cc55c5a4a00b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:59.954290 16267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748564520c0c4ebda80e6f8c7f9acd13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.954576 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748564520c0c4ebda80e6f8c7f9acd13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.954708 13579 raft_consensus.cc:2932] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:59.955399 16134 tablet_service.cc:1467] Processing CreateTablet for tablet a1c0b2fdd1d64bac8100d797c86fd8b2 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9dc6b07ff15645ab9222cc55c5a4a00b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:59.955814 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c0b2fdd1d64bac8100d797c86fd8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.957617 17519 raft_consensus.cc:1050] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 to VOTER
I20240502 14:14:59.957631 13315 raft_consensus.cc:1261] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:59.958345 17519 consensus_queue.cc:237] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:14:59.958554 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 748564520c0c4ebda80e6f8c7f9acd13 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9dc6b07ff15645ab9222cc55c5a4a00b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:59.958992 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748564520c0c4ebda80e6f8c7f9acd13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.959450 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 990da252c05c45d8b7a3e47288f1ab61 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9dc6b07ff15645ab9222cc55c5a4a00b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:59.959818 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 990da252c05c45d8b7a3e47288f1ab61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.960300 16265 tablet_service.cc:1467] Processing CreateTablet for tablet a1c0b2fdd1d64bac8100d797c86fd8b2 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9dc6b07ff15645ab9222cc55c5a4a00b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:59.960669 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c0b2fdd1d64bac8100d797c86fd8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.960665 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 990da252c05c45d8b7a3e47288f1ab61 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9dc6b07ff15645ab9222cc55c5a4a00b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:59.961031 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 990da252c05c45d8b7a3e47288f1ab61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.961606 16401 tablet_service.cc:1467] Processing CreateTablet for tablet a1c0b2fdd1d64bac8100d797c86fd8b2 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9dc6b07ff15645ab9222cc55c5a4a00b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:59.962006 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c0b2fdd1d64bac8100d797c86fd8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.965493 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 990da252c05c45d8b7a3e47288f1ab61 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=9dc6b07ff15645ab9222cc55c5a4a00b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:14:59.965866 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 990da252c05c45d8b7a3e47288f1ab61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.965764 17790 consensus_queue.cc:1035] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:59.970129 13045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:14:59.974248 13044 catalog_manager.cc:5484] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:14:59.982872 13317 raft_consensus.cc:1261] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:59.982959 13187 raft_consensus.cc:1261] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:59.983491 17576 consensus_queue.cc:1035] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:59.983639 16285 raft_consensus.cc:1261] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:14:59.984012 17576 consensus_queue.cc:1035] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:59.985397 16133 tablet_service.cc:1467] Processing CreateTablet for tablet 097e8add670f45aebdf6d955efdd69b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7c30cbfca4e14557a61f54bb79102113]), partition=
I20240502 14:14:59.985303 17086 consensus_queue.cc:1035] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:14:59.985666 16133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097e8add670f45aebdf6d955efdd69b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.986719 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 097e8add670f45aebdf6d955efdd69b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7c30cbfca4e14557a61f54bb79102113]), partition=
I20240502 14:14:59.986980 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097e8add670f45aebdf6d955efdd69b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.987099 17790 raft_consensus.cc:2932] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:14:59.990196 16002 tablet_service.cc:1467] Processing CreateTablet for tablet 097e8add670f45aebdf6d955efdd69b5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7c30cbfca4e14557a61f54bb79102113]), partition=
I20240502 14:14:59.990520 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 097e8add670f45aebdf6d955efdd69b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:14:59.995743 13445 consensus_queue.cc:237] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:00.001929 13580 raft_consensus.cc:2932] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:00.002293 13036 catalog_manager.cc:5484] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:00.002796 13317 raft_consensus.cc:2932] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:00.005126 17991 tablet_bootstrap.cc:492] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:00.007651 16420 consensus_queue.cc:237] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:00.013288 17993 tablet_bootstrap.cc:492] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:00.014272 13579 raft_consensus.cc:1261] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:00.014050 13188 raft_consensus.cc:2932] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:00.014825 17780 consensus_queue.cc:1035] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:00.017736 13319 raft_consensus.cc:1261] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:00.018131 17991 tablet_bootstrap.cc:654] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.018672 17780 consensus_queue.cc:1035] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:00.020555 13186 raft_consensus.cc:1261] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:00.023351 16288 raft_consensus.cc:1261] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:00.023658 17994 consensus_queue.cc:1035] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:00.023829 17780 raft_consensus.cc:2932] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:00.025300 17991 tablet_bootstrap.cc:492] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:00.025985 17993 tablet_bootstrap.cc:654] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.030874 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f961cf70d8ca482ba331241052833fd3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:00.033972 13317 raft_consensus.cc:2932] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:00.039034 17993 tablet_bootstrap.cc:492] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:00.039530 17993 ts_tablet_manager.cc:1367] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
I20240502 14:15:00.040629 17993 raft_consensus.cc:348] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:00.041584 17991 ts_tablet_manager.cc:1367] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.002s
I20240502 14:15:00.041603 13042 catalog_manager.cc:5484] T f961cf70d8ca482ba331241052833fd3 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:00.042671 17991 raft_consensus.cc:348] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:00.043112 17993 raft_consensus.cc:374] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.043368 17993 raft_consensus.cc:729] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.043871 17993 consensus_queue.cc:260] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:00.044479 17991 raft_consensus.cc:374] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.044638 17576 consensus_queue.cc:1035] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:00.044909 17991 raft_consensus.cc:729] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.045351 16133 tablet_service.cc:1514] Processing DeleteTablet for tablet f961cf70d8ca482ba331241052833fd3 with delete_type TABLET_DATA_TOMBSTONED (TS c86f7440570f4796babfd44c3d7fd847 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:00.045508 17991 consensus_queue.cc:260] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:00.048168 17991 ts_tablet_manager.cc:1398] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:15:00.050765 13579 raft_consensus.cc:2932] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:00.052469 17993 ts_tablet_manager.cc:1398] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20240502 14:15:00.053308 17991 tablet_bootstrap.cc:492] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:00.053681 17993 tablet_bootstrap.cc:492] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:00.054968 17991 tablet_bootstrap.cc:654] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.055785 17999 tablet_bootstrap.cc:492] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:00.057464 17999 tablet_bootstrap.cc:654] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.058328 17993 tablet_bootstrap.cc:654] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.060046 17576 raft_consensus.cc:2932] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:00.064193 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ab51b049c3674b629b55ed29df5b1fab with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:00.065121 17576 raft_consensus.cc:2932] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:00.070323 17993 tablet_bootstrap.cc:492] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:00.071307 17993 ts_tablet_manager.cc:1367] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240502 14:15:00.072122 16284 raft_consensus.cc:2932] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:00.083308 17990 tablet_bootstrap.cc:492] T 748564520c0c4ebda80e6f8c7f9acd13 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
W20240502 14:15:00.083873 13122 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.194:33193, user_credentials={real_user=slave}} blocked reactor thread for 83436.4us
I20240502 14:15:00.083058 18009 ts_tablet_manager.cc:902] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:00.086687 17993 raft_consensus.cc:348] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:00.088265 17990 tablet_bootstrap.cc:654] T 748564520c0c4ebda80e6f8c7f9acd13 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.090513 17993 raft_consensus.cc:374] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.091106 17993 raft_consensus.cc:729] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.088214 18009 tablet_copy_client.cc:308] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:15:00.091143 13317 raft_consensus.cc:2932] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:00.091825 17685 raft_consensus.cc:1050] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER d1da7f11f0604f78934354a992bb0f55 to VOTER
I20240502 14:15:00.091924 17993 consensus_queue.cc:260] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:00.092422 17993 ts_tablet_manager.cc:1398] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:15:00.092548 17685 consensus_queue.cc:237] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:00.093972 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:56671)
I20240502 14:15:00.094120 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:56671: session id = 8b47d4390f8343a493d6c00123785968-21ac2a9fa2154a0bbbe15aeaf178ea1a
I20240502 14:15:00.095144 17974 tablet_replica.cc:330] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:00.095391 17974 raft_consensus.cc:2229] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:00.095530 17974 pending_rounds.cc:62] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847: Trying to abort 1 pending ops.
I20240502 14:15:00.095642 17974 pending_rounds.cc:68] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023244492545499136 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "f961cf70d8ca482ba331241052833fd3" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } } }
I20240502 14:15:00.096099 17974 raft_consensus.cc:2894] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 14:15:00.096227 17974 raft_consensus.cc:2866] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:15:00.096318 17974 raft_consensus.cc:2258] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:00.096666 13339 tablet_copy_source_session.cc:205] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:00.099442 18009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.102962 13036 catalog_manager.cc:5484] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:00.103341 17988 tablet_bootstrap.cc:492] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:00.105094 17974 ts_tablet_manager.cc:1875] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:00.113422 17790 raft_consensus.cc:482] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:00.113612 17790 raft_consensus.cc:504] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:00.114737 17790 leader_election.cc:288] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:00.115267 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990da252c05c45d8b7a3e47288f1ab61" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
W20240502 14:15:00.116794 16354 leader_election.cc:341] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:00.117487 16284 raft_consensus.cc:1203] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:00.122853 17974 ts_tablet_manager.cc:1888] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:00.123024 17974 log.cc:1196] T f961cf70d8ca482ba331241052833fd3 P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/f961cf70d8ca482ba331241052833fd3
I20240502 14:15:00.125032 18009 tablet_copy_client.cc:783] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:00.125344 18009 tablet_copy_client.cc:647] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:00.126830 13042 catalog_manager.cc:5484] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 5 to 6, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:00.127154 13188 raft_consensus.cc:2932] T ab51b049c3674b629b55ed29df5b1fab P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:00.130520 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet ab51b049c3674b629b55ed29df5b1fab with delete_type TABLET_DATA_TOMBSTONED (TS 7eb4ecb2379b44c0954f2658c4a04350 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58526
I20240502 14:15:00.137292 13186 raft_consensus.cc:2932] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:00.149150 17753 raft_consensus.cc:1050] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847: attempting to promote NON_VOTER d1da7f11f0604f78934354a992bb0f55 to VOTER
I20240502 14:15:00.150216 17753 consensus_queue.cc:237] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:00.153429 13445 raft_consensus.cc:1261] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:00.154935 17705 consensus_queue.cc:1035] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:00.158768 16420 raft_consensus.cc:1261] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:00.159356 17753 consensus_queue.cc:1035] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:00.159751 13186 raft_consensus.cc:1261] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:00.160391 17705 consensus_queue.cc:1035] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:00.160990 13187 raft_consensus.cc:1261] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:00.161583 17753 consensus_queue.cc:1035] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:00.169981 17927 ts_tablet_manager.cc:902] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:00.170560 13447 consensus_queue.cc:237] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:00.173010 13188 raft_consensus.cc:1261] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:00.173498 18007 consensus_queue.cc:1035] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:00.177101 13187 raft_consensus.cc:2932] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:00.179919 17753 raft_consensus.cc:2932] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.181385 17086 raft_consensus.cc:2932] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:00.184902 13187 raft_consensus.cc:2932] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.185144 16420 raft_consensus.cc:2932] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.185658 13040 catalog_manager.cc:5484] T 536c22c444344bb485d3e522f0f77154 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:00.191500 13040 catalog_manager.cc:5484] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 2 to 3, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.196707 16265 tablet_service.cc:1514] Processing DeleteTablet for tablet 536c22c444344bb485d3e522f0f77154 with delete_type TABLET_DATA_TOMBSTONED (TS 5297e816f782430d9a01220f0471a882 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:00.197876 18023 tablet_replica.cc:330] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:00.198143 18023 raft_consensus.cc:2229] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:00.199030 18023 raft_consensus.cc:2258] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:00.199846 18023 ts_tablet_manager.cc:1875] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:00.203739 16284 raft_consensus.cc:2932] T ab51b049c3674b629b55ed29df5b1fab P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:00.204460 18023 ts_tablet_manager.cc:1888] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:00.204623 18023 log.cc:1196] T 536c22c444344bb485d3e522f0f77154 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/536c22c444344bb485d3e522f0f77154
I20240502 14:15:00.205451 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 536c22c444344bb485d3e522f0f77154 (table rebalancer_extra_table_0 [id=a6584ce357f343ffa6c92e7412b11020]) successfully deleted
I20240502 14:15:00.209703 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990da252c05c45d8b7a3e47288f1ab61" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:00.211259 17988 tablet_bootstrap.cc:654] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.214605 13318 raft_consensus.cc:1261] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:15:00.215821 16355 leader_election.cc:341] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:00.216022 16355 leader_election.cc:302] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b47d4390f8343a493d6c00123785968; no voters: c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:00.216748 18001 raft_consensus.cc:2726] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:00.216857 18009 tablet_copy_client.cc:523] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:00.218344 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet f961cf70d8ca482ba331241052833fd3 (table rebalancer_extra_table_2 [id=7a3f9a47775b46e38274ca5c7b2dbe6d]) successfully deleted
I20240502 14:15:00.151517 16021 raft_consensus.cc:1261] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:00.158432 16022 raft_consensus.cc:1261] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:00.218688 17986 consensus_queue.cc:1035] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:00.219266 17720 consensus_queue.cc:1035] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:00.220875 18009 tablet_bootstrap.cc:492] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:00.221642 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 536c22c444344bb485d3e522f0f77154 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:00.222184 17685 consensus_queue.cc:1035] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:00.225718 17999 tablet_bootstrap.cc:492] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:00.225857 17999 ts_tablet_manager.cc:1367] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.170s user 0.000s sys 0.002s
I20240502 14:15:00.226881 17999 raft_consensus.cc:348] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:00.224928 13318 raft_consensus.cc:2932] T 536c22c444344bb485d3e522f0f77154 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:00.227191 17999 raft_consensus.cc:374] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.227404 17999 raft_consensus.cc:729] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.227725 17999 consensus_queue.cc:260] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:00.230046 18015 tablet_replica.cc:330] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:15:00.230608 16022 raft_consensus.cc:2932] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.230283 18015 raft_consensus.cc:2229] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:00.232407 18015 raft_consensus.cc:2258] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:00.236091 17685 raft_consensus.cc:2932] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.238325 16152 raft_consensus.cc:1203] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:00.239193 18015 ts_tablet_manager.cc:1875] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:00.241566 13187 raft_consensus.cc:2932] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.244241 13445 raft_consensus.cc:2932] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.246132 13045 catalog_manager.cc:5484] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:00.250908 17988 tablet_bootstrap.cc:492] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:00.251096 17988 ts_tablet_manager.cc:1367] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.148s user 0.005s sys 0.005s
I20240502 14:15:00.251864 17988 raft_consensus.cc:348] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:00.251601 16022 raft_consensus.cc:2932] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.252103 17988 raft_consensus.cc:374] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.252229 17988 raft_consensus.cc:729] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.252481 17988 consensus_queue.cc:260] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:00.258677 17988 ts_tablet_manager.cc:1398] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 14:15:00.258731 13579 consensus_queue.cc:237] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:00.260191 18015 ts_tablet_manager.cc:1888] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:00.261853 17988 tablet_bootstrap.cc:492] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:00.262252 17991 tablet_bootstrap.cc:492] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:00.262413 17991 ts_tablet_manager.cc:1367] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.209s user 0.020s sys 0.023s
I20240502 14:15:00.263154 17991 raft_consensus.cc:348] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:00.263379 17991 raft_consensus.cc:374] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.263389 16021 raft_consensus.cc:1261] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:00.263463 17991 raft_consensus.cc:729] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.263682 17988 tablet_bootstrap.cc:654] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.271003 17999 ts_tablet_manager.cc:1398] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.031s user 0.001s sys 0.001s
I20240502 14:15:00.274981 17988 tablet_bootstrap.cc:492] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:00.275135 17988 ts_tablet_manager.cc:1367] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.001s
I20240502 14:15:00.275810 17988 raft_consensus.cc:348] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:00.276010 17988 raft_consensus.cc:374] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.276126 17988 raft_consensus.cc:729] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.276356 17988 consensus_queue.cc:260] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:00.276855 17988 ts_tablet_manager.cc:1398] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:00.278029 17999 tablet_bootstrap.cc:492] T 748564520c0c4ebda80e6f8c7f9acd13 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:00.279007 17990 tablet_bootstrap.cc:492] T 748564520c0c4ebda80e6f8c7f9acd13 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:00.279201 17990 ts_tablet_manager.cc:1367] T 748564520c0c4ebda80e6f8c7f9acd13 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.196s user 0.004s sys 0.000s
I20240502 14:15:00.280283 17990 raft_consensus.cc:348] T 748564520c0c4ebda80e6f8c7f9acd13 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:00.280560 17990 raft_consensus.cc:374] T 748564520c0c4ebda80e6f8c7f9acd13 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.280673 17990 raft_consensus.cc:729] T 748564520c0c4ebda80e6f8c7f9acd13 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.281008 17990 consensus_queue.cc:260] T 748564520c0c4ebda80e6f8c7f9acd13 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:00.281651 17990 ts_tablet_manager.cc:1398] T 748564520c0c4ebda80e6f8c7f9acd13 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:00.281986 17990 tablet_bootstrap.cc:492] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:00.283963 17999 tablet_bootstrap.cc:654] T 748564520c0c4ebda80e6f8c7f9acd13 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.286248 17990 tablet_bootstrap.cc:654] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.290999 13447 raft_consensus.cc:1261] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:00.296481 17990 tablet_bootstrap.cc:492] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:00.296646 17990 ts_tablet_manager.cc:1367] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20240502 14:15:00.267046 17991 consensus_queue.cc:260] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:00.296964 17999 tablet_bootstrap.cc:492] T 748564520c0c4ebda80e6f8c7f9acd13 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:00.297120 17999 ts_tablet_manager.cc:1367] T 748564520c0c4ebda80e6f8c7f9acd13 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240502 14:15:00.297384 18009 tablet_bootstrap.cc:492] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:00.297667 17990 raft_consensus.cc:348] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:00.297925 17990 raft_consensus.cc:374] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.297981 17991 ts_tablet_manager.cc:1398] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20240502 14:15:00.298135 17990 raft_consensus.cc:729] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.298441 17990 consensus_queue.cc:260] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:00.299146 17990 ts_tablet_manager.cc:1398] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:00.300249 18009 tablet_bootstrap.cc:492] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:15:00.300535 18009 ts_tablet_manager.cc:1367] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.080s user 0.035s sys 0.007s
I20240502 14:15:00.301321 18009 raft_consensus.cc:348] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:00.301563 18009 raft_consensus.cc:729] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:15:00.301743 18009 consensus_queue.cc:260] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:00.302929 18009 ts_tablet_manager.cc:1398] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:00.303499 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-21ac2a9fa2154a0bbbe15aeaf178ea1a received from {username='slave'} at 127.6.196.200:56671
I20240502 14:15:00.303598 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 8b47d4390f8343a493d6c00123785968-21ac2a9fa2154a0bbbe15aeaf178ea1a on tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:15:00.305804 17991 tablet_bootstrap.cc:492] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:00.310107 17991 tablet_bootstrap.cc:654] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.310961 17753 raft_consensus.cc:482] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:00.311213 17753 raft_consensus.cc:504] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:00.312199 17991 tablet_bootstrap.cc:492] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:00.312150 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990da252c05c45d8b7a3e47288f1ab61" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:00.312348 16420 raft_consensus.cc:2454] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:00.312376 17991 ts_tablet_manager.cc:1367] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240502 14:15:00.312426 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990da252c05c45d8b7a3e47288f1ab61" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:00.312636 13317 raft_consensus.cc:2454] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:00.312942 16089 leader_election.cc:302] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:00.313174 17753 leader_election.cc:288] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:00.313167 17991 raft_consensus.cc:348] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:00.313328 18022 raft_consensus.cc:2781] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:00.313401 17991 raft_consensus.cc:374] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.313481 17991 raft_consensus.cc:729] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.313405 18022 raft_consensus.cc:482] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:00.313616 18022 raft_consensus.cc:3037] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.313709 17991 consensus_queue.cc:260] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:00.314208 17991 ts_tablet_manager.cc:1398] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:00.315773 18022 raft_consensus.cc:504] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:00.317427 17999 raft_consensus.cc:348] T 748564520c0c4ebda80e6f8c7f9acd13 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:00.317662 17999 raft_consensus.cc:374] T 748564520c0c4ebda80e6f8c7f9acd13 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.317745 17999 raft_consensus.cc:729] T 748564520c0c4ebda80e6f8c7f9acd13 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.318022 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990da252c05c45d8b7a3e47288f1ab61" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:00.317993 17999 consensus_queue.cc:260] T 748564520c0c4ebda80e6f8c7f9acd13 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:00.318045 13317 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990da252c05c45d8b7a3e47288f1ab61" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:00.318219 16420 raft_consensus.cc:3037] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.318224 13317 raft_consensus.cc:3037] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.320871 16420 raft_consensus.cc:2454] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:00.320885 13317 raft_consensus.cc:2454] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:00.321326 16090 leader_election.cc:302] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:00.321573 18022 leader_election.cc:288] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:00.321722 18022 raft_consensus.cc:2781] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:00.322706 17999 ts_tablet_manager.cc:1398] T 748564520c0c4ebda80e6f8c7f9acd13 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240502 14:15:00.322777 18022 raft_consensus.cc:686] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:00.323036 18022 consensus_queue.cc:237] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:00.324685 17999 tablet_bootstrap.cc:492] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:00.325289 17927 tablet_copy_client.cc:308] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:15:00.328367 17999 tablet_bootstrap.cc:654] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:00.335467 13045 catalog_manager.cc:5484] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:00.263935 17685 consensus_queue.cc:1035] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:00.339783 18015 log.cc:1196] T ab51b049c3674b629b55ed29df5b1fab P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/ab51b049c3674b629b55ed29df5b1fab
I20240502 14:15:00.326037 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet 9ad2eb265e0f469787eb1699e339e0ed from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:42449)
I20240502 14:15:00.340620 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet 9ad2eb265e0f469787eb1699e339e0ed from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:42449: session id = c86f7440570f4796babfd44c3d7fd847-9ad2eb265e0f469787eb1699e339e0ed
I20240502 14:15:00.341423 17999 tablet_bootstrap.cc:492] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:00.341588 17999 ts_tablet_manager.cc:1367] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20240502 14:15:00.342384 17999 raft_consensus.cc:348] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:00.342584 17999 raft_consensus.cc:374] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:00.342681 17999 raft_consensus.cc:729] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:00.342931 17999 consensus_queue.cc:260] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:00.343272 13601 tablet_copy_source_session.cc:205] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:00.343500 17999 ts_tablet_manager.cc:1398] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:00.343729 17398 consensus_queue.cc:1035] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:00.347281 17398 raft_consensus.cc:2932] T db06293ca0914bb88dbff3808e3f1874 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.350548 16022 raft_consensus.cc:2932] T db06293ca0914bb88dbff3808e3f1874 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.352317 13447 raft_consensus.cc:2932] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.362527 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db06293ca0914bb88dbff3808e3f1874 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:00.362787 13045 catalog_manager.cc:5484] T db06293ca0914bb88dbff3808e3f1874 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.369590 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet db06293ca0914bb88dbff3808e3f1874 with delete_type TABLET_DATA_TOMBSTONED (TS 20a967bad5504b1e9c6fb720ecdd1625 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37690
I20240502 14:15:00.372938 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet ab51b049c3674b629b55ed29df5b1fab (table rebalance_test_table_17 [id=cbc2cd67f3c24195825b542cf2878195]) successfully deleted
I20240502 14:15:00.378160 17927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad2eb265e0f469787eb1699e339e0ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:00.402863 18051 tablet_replica.cc:330] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:15:00.403061 18051 raft_consensus.cc:2229] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:00.403285 18051 raft_consensus.cc:2258] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:00.404734 18051 ts_tablet_manager.cc:1875] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:00.406899 17927 tablet_copy_client.cc:783] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:00.407125 17927 tablet_copy_client.cc:647] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:00.409317 17927 tablet_copy_client.cc:523] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:00.416512 17927 tablet_bootstrap.cc:492] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:00.423405 17753 raft_consensus.cc:482] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:00.423566 17753 raft_consensus.cc:504] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:00.424201 17753 leader_election.cc:288] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:00.424705 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097e8add670f45aebdf6d955efdd69b5" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:00.424840 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097e8add670f45aebdf6d955efdd69b5" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:00.424904 16022 raft_consensus.cc:2454] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:00.425063 16284 raft_consensus.cc:2454] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:00.425316 16090 leader_election.cc:302] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:00.425729 17753 raft_consensus.cc:2781] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:00.425843 17753 raft_consensus.cc:482] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:00.425920 17753 raft_consensus.cc:3037] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.428223 17753 raft_consensus.cc:504] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:00.429003 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097e8add670f45aebdf6d955efdd69b5" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:00.429183 16022 raft_consensus.cc:3037] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.431815 16022 raft_consensus.cc:2454] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:00.432427 16090 leader_election.cc:302] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:00.433035 17753 leader_election.cc:288] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:00.433190 17753 raft_consensus.cc:2781] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:00.433310 17753 raft_consensus.cc:686] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:00.433401 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "097e8add670f45aebdf6d955efdd69b5" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:00.433604 16284 raft_consensus.cc:3037] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.433539 17753 consensus_queue.cc:237] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:00.441990 16284 raft_consensus.cc:2454] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:00.451239 13036 catalog_manager.cc:5484] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:00.456818 18051 ts_tablet_manager.cc:1888] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:00.473539 18051 log.cc:1196] T db06293ca0914bb88dbff3808e3f1874 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/db06293ca0914bb88dbff3808e3f1874
I20240502 14:15:00.474336 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet db06293ca0914bb88dbff3808e3f1874 (table rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]) successfully deleted
I20240502 14:15:00.501399 17514 raft_consensus.cc:482] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:00.501546 17514 raft_consensus.cc:504] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:00.502174 17514 leader_election.cc:288] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:00.503093 16285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748564520c0c4ebda80e6f8c7f9acd13" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:00.503288 16285 raft_consensus.cc:2454] T 748564520c0c4ebda80e6f8c7f9acd13 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:00.504173 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748564520c0c4ebda80e6f8c7f9acd13" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:00.504400 16022 raft_consensus.cc:2454] T 748564520c0c4ebda80e6f8c7f9acd13 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:00.504881 13253 leader_election.cc:302] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:00.505589 18054 raft_consensus.cc:2781] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:00.505684 18054 raft_consensus.cc:482] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:00.505789 18054 raft_consensus.cc:3037] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.506254 18001 raft_consensus.cc:482] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:00.506410 18001 raft_consensus.cc:504] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:00.507032 18001 leader_election.cc:288] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:00.507660 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c0b2fdd1d64bac8100d797c86fd8b2" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:00.507853 16152 raft_consensus.cc:2454] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:15:00.508414 18054 raft_consensus.cc:504] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:00.508677 16354 leader_election.cc:302] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:00.508952 18054 leader_election.cc:288] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:00.509163 18001 raft_consensus.cc:2781] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:00.509289 18001 raft_consensus.cc:482] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:00.509404 18001 raft_consensus.cc:3037] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.510876 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748564520c0c4ebda80e6f8c7f9acd13" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:00.511080 16284 raft_consensus.cc:3037] T 748564520c0c4ebda80e6f8c7f9acd13 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.511852 18001 raft_consensus.cc:504] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:00.512773 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c0b2fdd1d64bac8100d797c86fd8b2" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:00.512971 16152 raft_consensus.cc:3037] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.513648 16284 raft_consensus.cc:2454] T 748564520c0c4ebda80e6f8c7f9acd13 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:00.514322 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748564520c0c4ebda80e6f8c7f9acd13" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:00.514528 16022 raft_consensus.cc:3037] T 748564520c0c4ebda80e6f8c7f9acd13 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.515324 16152 raft_consensus.cc:2454] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:15:00.515676 16354 leader_election.cc:302] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:00.516258 17790 raft_consensus.cc:2781] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:00.516384 17790 raft_consensus.cc:686] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:15:00.516624 17790 consensus_queue.cc:237] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:00.517086 16022 raft_consensus.cc:2454] T 748564520c0c4ebda80e6f8c7f9acd13 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:00.518625 16285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c0b2fdd1d64bac8100d797c86fd8b2" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:00.518790 16285 raft_consensus.cc:2454] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:15:00.519002 13253 leader_election.cc:302] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:00.519800 17514 raft_consensus.cc:2781] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:00.519815 16285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1c0b2fdd1d64bac8100d797c86fd8b2" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:00.520006 17514 raft_consensus.cc:686] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:00.520035 16285 raft_consensus.cc:3037] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:00.520262 17514 consensus_queue.cc:237] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:00.522533 16285 raft_consensus.cc:2454] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:15:00.523072 18001 leader_election.cc:288] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:00.534456 13044 catalog_manager.cc:5484] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:00.541661 17927 tablet_bootstrap.cc:492] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:00.542279 17927 tablet_bootstrap.cc:492] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:15:00.542526 17927 ts_tablet_manager.cc:1367] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.126s user 0.056s sys 0.015s
I20240502 14:15:00.543563 17927 raft_consensus.cc:348] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:00.543875 17927 raft_consensus.cc:729] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:15:00.544164 17927 consensus_queue.cc:260] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:00.544901 17927 ts_tablet_manager.cc:1398] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240502 14:15:00.545630 13036 catalog_manager.cc:5484] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:00.547042 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-9ad2eb265e0f469787eb1699e339e0ed received from {username='slave'} at 127.6.196.198:42449
I20240502 14:15:00.547250 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-9ad2eb265e0f469787eb1699e339e0ed on tablet 9ad2eb265e0f469787eb1699e339e0ed with peer c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:00.584457 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:00.605479 16265 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:15:00.608099 16284 raft_consensus.cc:1261] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:00.608935 16022 raft_consensus.cc:1261] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:00.615569 16420 raft_consensus.cc:1203] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:00.616688 18022 consensus_queue.cc:1035] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:00.618911 17753 consensus_queue.cc:1035] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:00.622830 17398 raft_consensus.cc:1050] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER c86f7440570f4796babfd44c3d7fd847 to VOTER
I20240502 14:15:00.623517 17398 consensus_queue.cc:237] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:00.624014 13168 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240502 14:15:00.632258 16401 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:15:00.641906 16152 raft_consensus.cc:1261] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:00.642427 17685 consensus_queue.cc:1035] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:00.645480 13186 raft_consensus.cc:1261] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:00.646230 17939 consensus_queue.cc:1035] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:00.647717 13447 raft_consensus.cc:1261] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:00.650081 17939 consensus_queue.cc:1035] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:00.691052 17945 raft_consensus.cc:2932] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:00.691735 13447 raft_consensus.cc:2932] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:00.696393 13560 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:00.698684 13036 catalog_manager.cc:5484] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:00.700482 13186 raft_consensus.cc:2932] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:00.704721 13044 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet 097e8add670f45aebdf6d955efdd69b5 (table rebalancer_dynamic_table [id=7c30cbfca4e14557a61f54bb79102113]). Expected version 1 got 0
I20240502 14:15:00.706629 13042 catalog_manager.cc:5576] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695) does not have the latest schema for tablet 097e8add670f45aebdf6d955efdd69b5 (table rebalancer_dynamic_table [id=7c30cbfca4e14557a61f54bb79102113]). Expected version 1 got 0
I20240502 14:15:00.707139 16152 raft_consensus.cc:2932] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:00.712358 17794 raft_consensus.cc:1050] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER 5297e816f782430d9a01220f0471a882 to VOTER
I20240502 14:15:00.713173 17794 consensus_queue.cc:237] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:00.717208 13580 consensus_queue.cc:237] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:00.720351 13186 raft_consensus.cc:1261] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:00.721258 16285 raft_consensus.cc:1261] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:00.721762 13445 raft_consensus.cc:1261] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:00.721705 17939 consensus_queue.cc:1035] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:00.722126 17939 consensus_queue.cc:1035] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:00.723838 17939 consensus_queue.cc:1035] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:00.725621 16000 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:00.731963 13318 raft_consensus.cc:1261] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:00.732378 17939 consensus_queue.cc:1035] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:00.738674 16154 raft_consensus.cc:1261] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:00.739928 17939 consensus_queue.cc:1035] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:00.740446 17705 raft_consensus.cc:2932] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:00.741065 13186 raft_consensus.cc:2932] T 766f50e1e5a44c8ba211a446bcf86705 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:00.741515 16154 raft_consensus.cc:2932] T 766f50e1e5a44c8ba211a446bcf86705 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:00.745695 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 766f50e1e5a44c8ba211a446bcf86705 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:00.750491 13042 catalog_manager.cc:5484] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:00.752339 18096 tablet.cc:1722] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:00.752939 18096 tablet.cc:1620] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:00.757357 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 766f50e1e5a44c8ba211a446bcf86705 with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:00.757562 16134 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240502 14:15:00.759323 18097 tablet.cc:1722] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:00.759951 18097 tablet.cc:1620] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:15:00.762514 13044 catalog_manager.cc:5576] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695) does not have the latest schema for tablet 097e8add670f45aebdf6d955efdd69b5 (table rebalancer_dynamic_table [id=7c30cbfca4e14557a61f54bb79102113]). Expected version 1 got 0
I20240502 14:15:00.767680 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=7c30cbfca4e14557a61f54bb79102113] alter complete (version 1)
I20240502 14:15:00.784904 13429 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:00.785933 16154 raft_consensus.cc:1203] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:00.804358 18100 tablet_replica.cc:330] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:00.809020 17705 raft_consensus.cc:2932] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:00.810639 13318 raft_consensus.cc:2932] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:00.810082 18098 tablet.cc:1722] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:00.812243 18098 tablet.cc:1620] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:00.814713 16284 raft_consensus.cc:2932] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:00.815752 18100 raft_consensus.cc:2229] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:00.817795 18100 raft_consensus.cc:2258] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:00.819298 18100 ts_tablet_manager.cc:1875] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:00.820842 13044 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet 097e8add670f45aebdf6d955efdd69b5 (table rebalancer_dynamic_table [id=7c30cbfca4e14557a61f54bb79102113]). Expected version 1 got 0
I20240502 14:15:00.821542 13036 catalog_manager.cc:5484] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:00.819089 13447 raft_consensus.cc:2932] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:00.829491 13297 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:00.835137 13580 consensus_queue.cc:237] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:00.837509 13447 raft_consensus.cc:1261] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:00.838301 17939 consensus_queue.cc:1035] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:00.840298 16154 consensus_queue.cc:237] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:00.842270 13186 raft_consensus.cc:1261] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:00.842370 16021 raft_consensus.cc:1261] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:00.842775 17720 consensus_queue.cc:1035] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:00.842913 17939 raft_consensus.cc:2932] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:00.843533 17720 consensus_queue.cc:1035] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:00.845507 16288 raft_consensus.cc:1261] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:00.845674 17720 raft_consensus.cc:2932] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.845991 17705 consensus_queue.cc:1035] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:00.846115 13447 raft_consensus.cc:2932] T 80040c60b029481285c50171bcc5ce35 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:00.846930 17753 consensus_queue.cc:1035] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:00.847163 16284 raft_consensus.cc:2932] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:00.848413 13186 raft_consensus.cc:2932] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.851092 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1e2b8c72c7ba4be58f31ac073e41f2a8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:00.851421 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 80040c60b029481285c50171bcc5ce35 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:00.854223 13046 catalog_manager.cc:5484] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:00.870811 13040 catalog_manager.cc:5484] T 80040c60b029481285c50171bcc5ce35 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:00.871382 17720 consensus_queue.cc:1035] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:00.876420 16021 raft_consensus.cc:2932] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:00.883361 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet 80040c60b029481285c50171bcc5ce35 with delete_type TABLET_DATA_TOMBSTONED (TS cd67493f6f0142f896010f2b8a39887d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57492
I20240502 14:15:00.884120 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet 1e2b8c72c7ba4be58f31ac073e41f2a8 with delete_type TABLET_DATA_TOMBSTONED (TS 8b47d4390f8343a493d6c00123785968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:00.886459 18100 ts_tablet_manager.cc:1888] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:00.891150 18107 tablet_replica.cc:330] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:00.892596 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:00.894071 18100 log.cc:1196] T 766f50e1e5a44c8ba211a446bcf86705 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/766f50e1e5a44c8ba211a446bcf86705
I20240502 14:15:00.899658 18106 tablet_replica.cc:330] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:15:00.900000 18106 raft_consensus.cc:2229] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:00.900101 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 766f50e1e5a44c8ba211a446bcf86705 (table rebalance_test_table_17 [id=cbc2cd67f3c24195825b542cf2878195]) successfully deleted
I20240502 14:15:00.900591 18106 raft_consensus.cc:2258] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:00.901485 18106 ts_tablet_manager.cc:1875] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:00.908210 18096 tablet.cc:1722] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:00.908778 18096 tablet.cc:1620] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:00.910856 18097 tablet.cc:1722] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:00.911183 18107 raft_consensus.cc:2229] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:00.911258 18097 tablet.cc:1620] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:15:00.912410 18107 raft_consensus.cc:2258] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:00.915223 18106 ts_tablet_manager.cc:1888] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:00.918382 18107 ts_tablet_manager.cc:1875] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:00.919957 18106 log.cc:1196] T 80040c60b029481285c50171bcc5ce35 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/80040c60b029481285c50171bcc5ce35
I20240502 14:15:00.921779 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 80040c60b029481285c50171bcc5ce35 (table rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]) successfully deleted
I20240502 14:15:00.926365 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=7c30cbfca4e14557a61f54bb79102113] alter complete (version 2)
I20240502 14:15:00.938011 18098 tablet.cc:1722] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:00.938730 18098 tablet.cc:1620] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:00.964478 18107 ts_tablet_manager.cc:1888] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:00.964669 18107 log.cc:1196] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/1e2b8c72c7ba4be58f31ac073e41f2a8
I20240502 14:15:00.967324 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 1e2b8c72c7ba4be58f31ac073e41f2a8 (table rebalancer_extra_table_2 [id=7a3f9a47775b46e38274ca5c7b2dbe6d]) successfully deleted
I20240502 14:15:01.004328 17790 consensus_queue.cc:1035] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.011346 17786 consensus_queue.cc:1035] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.018769 17940 raft_consensus.cc:1050] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:15:01.019690 17940 consensus_queue.cc:237] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:01.025264 13580 raft_consensus.cc:1261] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:01.025319 13447 raft_consensus.cc:1261] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:01.026129 16419 raft_consensus.cc:1261] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:01.026702 17940 consensus_queue.cc:1035] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:01.027359 18113 consensus_queue.cc:1035] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:01.028045 18113 consensus_queue.cc:1035] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:01.030946 13046 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:01.031711 18054 raft_consensus.cc:2932] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:01.035241 17514 consensus_queue.cc:1035] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.037477 13580 raft_consensus.cc:2932] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:01.039048 13447 raft_consensus.cc:2932] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:01.040941 16419 raft_consensus.cc:2932] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:01.046028 18096 tablet.cc:1722] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:01.048100 13042 catalog_manager.cc:5484] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:01.049999 17940 consensus_queue.cc:1035] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.056242 18097 tablet.cc:1722] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:01.060393 13040 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=7c30cbfca4e14557a61f54bb79102113] alter complete (version 3)
I20240502 14:15:01.067193 13036 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:01.068171 13317 consensus_queue.cc:237] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:01.070420 13579 raft_consensus.cc:1261] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:01.071094 17514 consensus_queue.cc:1035] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:01.071727 18098 tablet.cc:1722] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:01.073194 16419 raft_consensus.cc:1261] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:01.076438 17940 raft_consensus.cc:2932] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:01.078743 18054 consensus_queue.cc:1035] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:01.080227 16419 raft_consensus.cc:2932] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:01.082684 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:01.085759 16401 tablet_service.cc:1467] Processing CreateTablet for tablet c28e61290d744d4c9409947f9e41706e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=35b2e9aa3c11478e9023ef811af72c1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.086139 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28e61290d744d4c9409947f9e41706e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.086372 16000 tablet_service.cc:1467] Processing CreateTablet for tablet c28e61290d744d4c9409947f9e41706e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=35b2e9aa3c11478e9023ef811af72c1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.086860 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28e61290d744d4c9409947f9e41706e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.087304 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 551bf8a880ec4157870bc449c0c54cf9 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=35b2e9aa3c11478e9023ef811af72c1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.087690 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551bf8a880ec4157870bc449c0c54cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.089932 16265 tablet_service.cc:1467] Processing CreateTablet for tablet c28e61290d744d4c9409947f9e41706e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=35b2e9aa3c11478e9023ef811af72c1a]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.090312 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28e61290d744d4c9409947f9e41706e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.094359 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb770f077c84cb99985ef59253f133b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=35b2e9aa3c11478e9023ef811af72c1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.094794 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb770f077c84cb99985ef59253f133b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.097373 18123 tablet_bootstrap.cc:492] T 551bf8a880ec4157870bc449c0c54cf9 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:01.097544 16400 tablet_service.cc:1467] Processing CreateTablet for tablet 551bf8a880ec4157870bc449c0c54cf9 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=35b2e9aa3c11478e9023ef811af72c1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.097857 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551bf8a880ec4157870bc449c0c54cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.099622 18123 tablet_bootstrap.cc:654] T 551bf8a880ec4157870bc449c0c54cf9 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.101895 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb770f077c84cb99985ef59253f133b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=35b2e9aa3c11478e9023ef811af72c1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.102279 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb770f077c84cb99985ef59253f133b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.105613 16002 tablet_service.cc:1467] Processing CreateTablet for tablet 2cb770f077c84cb99985ef59253f133b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=35b2e9aa3c11478e9023ef811af72c1a]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.106015 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb770f077c84cb99985ef59253f133b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.106074 16003 tablet_service.cc:1467] Processing CreateTablet for tablet 551bf8a880ec4157870bc449c0c54cf9 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=35b2e9aa3c11478e9023ef811af72c1a]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:01.106516 16003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 551bf8a880ec4157870bc449c0c54cf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.112624 13036 catalog_manager.cc:5484] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:01.117612 13578 raft_consensus.cc:2932] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:01.120402 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:01.120817 18100 tablet_replica.cc:330] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:01.121094 18100 raft_consensus.cc:2229] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:01.121372 18100 raft_consensus.cc:2258] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:01.127076 18100 ts_tablet_manager.cc:1875] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:01.173461 13040 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:01.173820 13040 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:01.178030 13040 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 097e8add670f45aebdf6d955efdd69b5
I20240502 14:15:01.178871 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet 097e8add670f45aebdf6d955efdd69b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:01 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:01.179472 15999 tablet_service.cc:1514] Processing DeleteTablet for tablet 097e8add670f45aebdf6d955efdd69b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:01 UTC) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:01.180063 16265 tablet_service.cc:1514] Processing DeleteTablet for tablet 097e8add670f45aebdf6d955efdd69b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:01 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:01.192448 18100 ts_tablet_manager.cc:1888] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:01.198542 18133 tablet_bootstrap.cc:492] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:01.201120 18133 tablet_bootstrap.cc:654] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.202345 18100 log.cc:1196] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/21ac2a9fa2154a0bbbe15aeaf178ea1a
I20240502 14:15:01.203445 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a (table rebalance_test_table_3 [id=8fa288258a0f4231be6de034088616ab]) successfully deleted
I20240502 14:15:01.203605 18128 tablet_bootstrap.cc:492] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:01.208201 18128 tablet_bootstrap.cc:654] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.209756 18130 tablet_replica.cc:330] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:01.210101 18130 raft_consensus.cc:2229] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:01.210433 18130 raft_consensus.cc:2258] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:01.211442 18130 ts_tablet_manager.cc:1875] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:01.211550 18133 tablet_bootstrap.cc:492] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:01.211670 18133 ts_tablet_manager.cc:1367] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20240502 14:15:01.212363 18133 raft_consensus.cc:348] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:01.212543 18133 raft_consensus.cc:374] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.212630 18133 raft_consensus.cc:729] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.214643 18133 consensus_queue.cc:260] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:01.215576 18133 ts_tablet_manager.cc:1398] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240502 14:15:01.216357 18133 tablet_bootstrap.cc:492] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:01.216408 18127 tablet_bootstrap.cc:492] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:01.219081 18131 tablet_replica.cc:330] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:01.219309 18133 tablet_bootstrap.cc:654] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.219470 18131 raft_consensus.cc:2229] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:01.228580 18133 tablet_bootstrap.cc:492] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:01.229055 18127 tablet_bootstrap.cc:654] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.229302 18133 ts_tablet_manager.cc:1367] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.005s
I20240502 14:15:01.230337 18133 raft_consensus.cc:348] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:01.231781 18133 raft_consensus.cc:374] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.219808 18131 raft_consensus.cc:2258] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:01.232040 18133 raft_consensus.cc:729] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.232530 18133 consensus_queue.cc:260] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:01.232813 18131 ts_tablet_manager.cc:1875] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:01.234309 18133 ts_tablet_manager.cc:1398] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:15:01.234550 18133 tablet_bootstrap.cc:492] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:01.236001 18133 tablet_bootstrap.cc:654] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.249579 18129 tablet_replica.cc:330] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:01.250560 18128 tablet_bootstrap.cc:492] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:01.250804 18129 raft_consensus.cc:2229] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:01.253506 18128 ts_tablet_manager.cc:1367] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.002s
I20240502 14:15:01.254657 18128 raft_consensus.cc:348] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:01.255064 18123 tablet_bootstrap.cc:492] T 551bf8a880ec4157870bc449c0c54cf9 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:01.255236 18129 raft_consensus.cc:2258] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:01.255295 18123 ts_tablet_manager.cc:1367] T 551bf8a880ec4157870bc449c0c54cf9 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.158s user 0.048s sys 0.056s
I20240502 14:15:01.256371 18128 raft_consensus.cc:374] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.256336 18123 raft_consensus.cc:348] T 551bf8a880ec4157870bc449c0c54cf9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:01.256597 18123 raft_consensus.cc:374] T 551bf8a880ec4157870bc449c0c54cf9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.256597 18129 ts_tablet_manager.cc:1875] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:01.256513 18128 raft_consensus.cc:729] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.256747 18123 raft_consensus.cc:729] T 551bf8a880ec4157870bc449c0c54cf9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.257010 18128 consensus_queue.cc:260] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:01.257041 18123 consensus_queue.cc:260] T 551bf8a880ec4157870bc449c0c54cf9 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:01.257800 18123 ts_tablet_manager.cc:1398] T 551bf8a880ec4157870bc449c0c54cf9 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:01.258010 17570 raft_consensus.cc:482] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:01.258176 17570 raft_consensus.cc:504] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:01.258816 17570 leader_election.cc:288] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:01.259559 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551bf8a880ec4157870bc449c0c54cf9" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:01.259701 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551bf8a880ec4157870bc449c0c54cf9" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:01.259958 13187 raft_consensus.cc:2454] T 551bf8a880ec4157870bc449c0c54cf9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
W20240502 14:15:01.260316 15956 leader_election.cc:341] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:01.260511 15955 leader_election.cc:302] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:01.261034 17570 raft_consensus.cc:2781] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:01.261175 17570 raft_consensus.cc:482] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:01.261289 17570 raft_consensus.cc:3037] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.263676 17570 raft_consensus.cc:504] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:01.264605 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551bf8a880ec4157870bc449c0c54cf9" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
W20240502 14:15:01.265036 15956 leader_election.cc:341] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:01.265694 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551bf8a880ec4157870bc449c0c54cf9" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:01.265877 13187 raft_consensus.cc:3037] T 551bf8a880ec4157870bc449c0c54cf9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.267153 18128 ts_tablet_manager.cc:1398] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240502 14:15:01.267866 18127 tablet_bootstrap.cc:492] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:01.267989 18127 ts_tablet_manager.cc:1367] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20240502 14:15:01.268666 13187 raft_consensus.cc:2454] T 551bf8a880ec4157870bc449c0c54cf9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:01.268813 18127 raft_consensus.cc:348] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:01.269016 18127 raft_consensus.cc:374] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.269162 18127 raft_consensus.cc:729] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.269215 15955 leader_election.cc:302] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:01.269619 17570 leader_election.cc:288] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:01.269814 17570 raft_consensus.cc:2781] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:01.269649 18127 consensus_queue.cc:260] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:01.269933 17570 raft_consensus.cc:686] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:01.270182 17570 consensus_queue.cc:237] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:01.275879 18130 ts_tablet_manager.cc:1888] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:01.276057 18130 log.cc:1196] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/097e8add670f45aebdf6d955efdd69b5
I20240502 14:15:01.276715 18130 ts_tablet_manager.cc:1909] T 097e8add670f45aebdf6d955efdd69b5 P d1da7f11f0604f78934354a992bb0f55: Deleting consensus metadata
I20240502 14:15:01.277217 18127 ts_tablet_manager.cc:1398] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:15:01.279064 18127 tablet_bootstrap.cc:492] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:01.281715 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:01.282215 18127 tablet_bootstrap.cc:654] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.282709 17786 raft_consensus.cc:482] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:01.282828 17786 raft_consensus.cc:504] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:01.283481 17786 leader_election.cc:288] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:01.284826 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551bf8a880ec4157870bc449c0c54cf9" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:01.285157 13187 raft_consensus.cc:2379] T 551bf8a880ec4157870bc449c0c54cf9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8b47d4390f8343a493d6c00123785968 in current term 1: Already voted for candidate d1da7f11f0604f78934354a992bb0f55 in this term.
I20240502 14:15:01.291837 18129 ts_tablet_manager.cc:1888] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:01.294339 16400 tablet_service.cc:1467] Processing CreateTablet for tablet c2b5c6a57c10481a8a351de00b1a6a76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4b9bb97f97c7437abbfb54275bfd92ea]), partition=
I20240502 14:15:01.294375 13427 tablet_service.cc:1467] Processing CreateTablet for tablet c2b5c6a57c10481a8a351de00b1a6a76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4b9bb97f97c7437abbfb54275bfd92ea]), partition=
I20240502 14:15:01.294632 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b5c6a57c10481a8a351de00b1a6a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.294857 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b5c6a57c10481a8a351de00b1a6a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.295771 16265 tablet_service.cc:1467] Processing CreateTablet for tablet c2b5c6a57c10481a8a351de00b1a6a76 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4b9bb97f97c7437abbfb54275bfd92ea]), partition=
I20240502 14:15:01.296080 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2b5c6a57c10481a8a351de00b1a6a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.302879 18128 tablet_bootstrap.cc:492] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:01.304132 17794 raft_consensus.cc:1050] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER c86f7440570f4796babfd44c3d7fd847 to VOTER
I20240502 14:15:01.304963 17794 consensus_queue.cc:237] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:01.312322 18131 ts_tablet_manager.cc:1888] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:01.312664 18131 log.cc:1196] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/097e8add670f45aebdf6d955efdd69b5
I20240502 14:15:01.316430 13445 raft_consensus.cc:1261] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:01.317339 17951 consensus_queue.cc:1035] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:01.320080 13187 raft_consensus.cc:1261] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:01.320828 17951 consensus_queue.cc:1035] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:01.323333 18141 tablet_bootstrap.cc:492] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:01.328007 18141 tablet_bootstrap.cc:654] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.332173 18128 tablet_bootstrap.cc:654] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.332975 17794 raft_consensus.cc:2932] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:01.334522 13046 catalog_manager.cc:5484] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:01.335683 18131 ts_tablet_manager.cc:1909] T 097e8add670f45aebdf6d955efdd69b5 P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:15:01.338174 13186 raft_consensus.cc:2932] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:01.338979 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 097e8add670f45aebdf6d955efdd69b5 (table rebalancer_dynamic_table_new_name [id=7c30cbfca4e14557a61f54bb79102113]) successfully deleted
I20240502 14:15:01.340358 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet 097e8add670f45aebdf6d955efdd69b5 (table rebalancer_dynamic_table_new_name [id=7c30cbfca4e14557a61f54bb79102113]) successfully deleted
W20240502 14:15:01.328143 15955 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.196.193:36687, user_credentials={real_user=slave}} blocked reactor thread for 58920.9us
I20240502 14:15:01.341805 13445 raft_consensus.cc:2932] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:01.344012 18139 raft_consensus.cc:482] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:01.344187 18139 raft_consensus.cc:504] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:01.345062 18139 leader_election.cc:288] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:01.344926 13040 catalog_manager.cc:5484] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:01.345589 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28e61290d744d4c9409947f9e41706e" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:01.346174 18133 tablet_bootstrap.cc:492] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
W20240502 14:15:01.346149 15957 leader_election.cc:341] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:01.346282 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28e61290d744d4c9409947f9e41706e" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:01.346503 18133 ts_tablet_manager.cc:1367] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.001s
W20240502 14:15:01.346871 15956 leader_election.cc:341] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:01.346977 15956 leader_election.cc:302] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:01.347923 18139 raft_consensus.cc:2726] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:01.347923 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "551bf8a880ec4157870bc449c0c54cf9" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:01.349175 18133 raft_consensus.cc:348] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:01.349417 18133 raft_consensus.cc:374] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.349500 18133 raft_consensus.cc:729] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.349705 18133 consensus_queue.cc:260] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:01.350349 18133 ts_tablet_manager.cc:1398] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:15:01.355026 18128 tablet_bootstrap.cc:492] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:01.355204 18128 ts_tablet_manager.cc:1367] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.003s
I20240502 14:15:01.355244 16355 leader_election.cc:302] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b47d4390f8343a493d6c00123785968; no voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:01.355325 13317 consensus_queue.cc:237] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:01.355741 17790 raft_consensus.cc:3037] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.355938 18128 raft_consensus.cc:348] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:01.356128 18128 raft_consensus.cc:374] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.315392 16154 raft_consensus.cc:1261] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:01.357133 17794 consensus_queue.cc:1035] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:01.358309 17790 raft_consensus.cc:2726] T 551bf8a880ec4157870bc449c0c54cf9 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:01.361371 18141 tablet_bootstrap.cc:492] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:01.361534 18141 ts_tablet_manager.cc:1367] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20240502 14:15:01.362422 18141 raft_consensus.cc:348] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:01.362659 18141 raft_consensus.cc:374] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.362771 18141 raft_consensus.cc:729] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.363013 18141 consensus_queue.cc:260] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:01.363646 18141 ts_tablet_manager.cc:1398] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:01.365660 18129 log.cc:1196] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/097e8add670f45aebdf6d955efdd69b5
I20240502 14:15:01.366351 18129 ts_tablet_manager.cc:1909] T 097e8add670f45aebdf6d955efdd69b5 P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:15:01.368407 18127 tablet_bootstrap.cc:492] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:01.356187 18128 raft_consensus.cc:729] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
W20240502 14:15:01.369033 13251 consensus_peers.cc:485] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d -> Peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Couldn't send request to peer 7eb4ecb2379b44c0954f2658c4a04350. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 819e6e83e6064c539cfc73a28c43db5b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:01.370074 18127 ts_tablet_manager.cc:1367] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.002s
I20240502 14:15:01.370440 16288 raft_consensus.cc:1261] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:01.370572 16022 raft_consensus.cc:1261] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:01.370666 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 097e8add670f45aebdf6d955efdd69b5 (table rebalancer_dynamic_table_new_name [id=7c30cbfca4e14557a61f54bb79102113]) successfully deleted
I20240502 14:15:01.371089 17940 consensus_queue.cc:1035] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:01.371515 17940 consensus_queue.cc:1035] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:01.372125 18104 raft_consensus.cc:482] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:01.372246 18104 raft_consensus.cc:504] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:01.372926 18104 leader_election.cc:288] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:01.384436 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb770f077c84cb99985ef59253f133b" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:01.384624 16021 raft_consensus.cc:2454] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:01.385052 16221 leader_election.cc:302] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:01.385403 18104 raft_consensus.cc:2781] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:01.385521 18104 raft_consensus.cc:482] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:01.385614 18104 raft_consensus.cc:3037] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.387164 17514 raft_consensus.cc:2932] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:01.388160 18104 raft_consensus.cc:504] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:01.389312 18104 leader_election.cc:288] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:01.390302 16284 raft_consensus.cc:2932] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:01.391921 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb770f077c84cb99985ef59253f133b" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:01.392076 16021 raft_consensus.cc:3037] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.394480 16021 raft_consensus.cc:2454] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:01.395350 16221 leader_election.cc:302] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:01.395715 17813 raft_consensus.cc:2781] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:01.395857 17813 raft_consensus.cc:686] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:01.396111 17813 consensus_queue.cc:237] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:01.400028 13040 catalog_manager.cc:5484] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:01.410343 16154 raft_consensus.cc:2932] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:01.374454 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb770f077c84cb99985ef59253f133b" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:01.422748 18127 raft_consensus.cc:348] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:01.389087 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb770f077c84cb99985ef59253f133b" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:01.422956 18127 raft_consensus.cc:374] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.423025 18127 raft_consensus.cc:729] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.423226 18127 consensus_queue.cc:260] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:01.369431 18128 consensus_queue.cc:260] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
W20240502 14:15:01.423774 16220 leader_election.cc:341] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:15:01.424073 16220 leader_election.cc:341] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:01.428463 13042 catalog_manager.cc:5484] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:01.434732 18128 ts_tablet_manager.cc:1398] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.079s user 0.001s sys 0.002s
I20240502 14:15:01.437906 18128 tablet_bootstrap.cc:492] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:01.442554 18128 tablet_bootstrap.cc:654] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.444434 18127 ts_tablet_manager.cc:1398] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.022s user 0.001s sys 0.000s
I20240502 14:15:01.444746 18127 tablet_bootstrap.cc:492] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:01.448158 18128 tablet_bootstrap.cc:492] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:01.448264 18128 ts_tablet_manager.cc:1367] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20240502 14:15:01.449153 18127 tablet_bootstrap.cc:654] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.456975 18127 tablet_bootstrap.cc:492] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:01.457149 18127 ts_tablet_manager.cc:1367] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20240502 14:15:01.457921 18127 raft_consensus.cc:348] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:01.458201 18127 raft_consensus.cc:374] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.458313 18127 raft_consensus.cc:729] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.458590 18128 raft_consensus.cc:348] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:01.458578 18127 consensus_queue.cc:260] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:01.458791 18128 raft_consensus.cc:374] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.458873 18128 raft_consensus.cc:729] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.459035 18128 consensus_queue.cc:260] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:01.459517 18128 ts_tablet_manager.cc:1398] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.011s user 0.000s sys 0.000s
I20240502 14:15:01.466338 18127 ts_tablet_manager.cc:1398] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:15:01.468397 17813 raft_consensus.cc:482] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:01.468541 17813 raft_consensus.cc:504] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:01.469238 17813 leader_election.cc:288] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:01.469941 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b5c6a57c10481a8a351de00b1a6a76" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:01.470010 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b5c6a57c10481a8a351de00b1a6a76" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:01.470152 13445 raft_consensus.cc:2454] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:01.470556 16219 leader_election.cc:302] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
W20240502 14:15:01.470707 16220 leader_election.cc:341] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:01.470009 18127 tablet_bootstrap.cc:492] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:01.470880 17813 raft_consensus.cc:2781] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:01.470973 17813 raft_consensus.cc:482] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:01.471041 17813 raft_consensus.cc:3037] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.472419 18127 tablet_bootstrap.cc:654] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:01.475550 17813 raft_consensus.cc:504] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:01.476466 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b5c6a57c10481a8a351de00b1a6a76" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:01.476625 13445 raft_consensus.cc:3037] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.478120 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b5c6a57c10481a8a351de00b1a6a76" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
W20240502 14:15:01.478605 16220 leader_election.cc:341] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:01.477629 17813 leader_election.cc:288] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:01.479343 13445 raft_consensus.cc:2454] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:01.480516 16219 leader_election.cc:302] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:01.482892 17813 raft_consensus.cc:2781] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:01.483248 17813 raft_consensus.cc:686] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:01.483502 17813 consensus_queue.cc:237] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:01.491554 18127 tablet_bootstrap.cc:492] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:01.491748 18127 ts_tablet_manager.cc:1367] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.003s
I20240502 14:15:01.492249 13447 consensus_queue.cc:237] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:01.492448 18127 raft_consensus.cc:348] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:01.492203 13040 catalog_manager.cc:5484] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:01.492605 18127 raft_consensus.cc:374] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:01.492655 18127 raft_consensus.cc:729] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:01.492830 18127 consensus_queue.cc:260] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:01.493266 18127 ts_tablet_manager.cc:1398] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:15:01.497429 16022 raft_consensus.cc:2932] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:01.498773 16154 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:01.499325 17086 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:01.500751 13186 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:15:01.503235 13382 consensus_peers.cc:485] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Couldn't send request to peer 7eb4ecb2379b44c0954f2658c4a04350. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7852539de07744caaf385620c5166861. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:01.503684 17086 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:01.506376 17986 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:01.507333 16154 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:01.510460 13186 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:01.513361 13040 catalog_manager.cc:5484] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 6 to 7, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:01.529906 16420 consensus_queue.cc:237] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:01.532644 16154 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:01.532640 16288 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:01.532874 16355 consensus_peers.cc:485] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b34f2a22306459784a1ad7b81cd1683. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:01.533147 17786 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:01.533603 17790 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:01.538621 17790 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:01.539220 16154 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:01.542932 17786 raft_consensus.cc:482] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:01.543079 17786 raft_consensus.cc:504] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:01.543709 17786 leader_election.cc:288] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:01.543992 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b5c6a57c10481a8a351de00b1a6a76" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:01.544251 13040 catalog_manager.cc:5484] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:01.544557 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2b5c6a57c10481a8a351de00b1a6a76" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:01.544791 13447 raft_consensus.cc:2379] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8b47d4390f8343a493d6c00123785968 in current term 1: Already voted for candidate 5297e816f782430d9a01220f0471a882 in this term.
I20240502 14:15:01.545163 16353 leader_election.cc:302] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b47d4390f8343a493d6c00123785968; no voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:15:01.545455 17786 raft_consensus.cc:3037] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.551585 17786 raft_consensus.cc:2726] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:01.557546 16288 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:01.567029 13578 consensus_queue.cc:237] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:01.569273 13447 raft_consensus.cc:1261] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:01.569430 13186 raft_consensus.cc:1261] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:15:01.569458 13516 consensus_peers.cc:485] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fa51110016d4cafafbe25a1bd213f2e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:01.569860 17939 consensus_queue.cc:1035] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:01.570334 17794 consensus_queue.cc:1035] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:01.572343 17939 raft_consensus.cc:2932] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:01.572953 13447 raft_consensus.cc:2932] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:01.573132 13186 raft_consensus.cc:2932] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:01.577477 18175 raft_consensus.cc:482] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:01.577611 18175 raft_consensus.cc:504] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:01.578295 18175 leader_election.cc:288] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:01.578624 13040 catalog_manager.cc:5484] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:01.580286 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28e61290d744d4c9409947f9e41706e" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:01.580508 16288 raft_consensus.cc:2454] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:15:01.580832 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28e61290d744d4c9409947f9e41706e" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:01.581056 16022 raft_consensus.cc:2454] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:15:01.581454 16355 leader_election.cc:302] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:01.581686 18175 raft_consensus.cc:2781] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:01.581794 18175 raft_consensus.cc:482] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:01.581884 18175 raft_consensus.cc:3037] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.584163 18175 raft_consensus.cc:504] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:01.584985 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28e61290d744d4c9409947f9e41706e" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:01.585250 16288 raft_consensus.cc:3037] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.585717 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c28e61290d744d4c9409947f9e41706e" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:01.585908 16022 raft_consensus.cc:3037] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.588133 16288 raft_consensus.cc:2454] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:15:01.588502 16355 leader_election.cc:302] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:01.588532 16022 raft_consensus.cc:2454] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:15:01.588752 17786 raft_consensus.cc:2781] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:01.588902 17786 raft_consensus.cc:686] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:15:01.589114 18175 leader_election.cc:288] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:01.589185 17786 consensus_queue.cc:237] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:01.591814 13036 catalog_manager.cc:5484] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:01.601006 13186 consensus_queue.cc:237] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:01.603693 13447 raft_consensus.cc:1261] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:15:01.603700 13122 consensus_peers.cc:485] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e51feadc94c48e9bcecffecacbda4b9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:01.604177 18195 consensus_queue.cc:1035] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:01.604667 13578 raft_consensus.cc:1261] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:01.605444 17453 consensus_queue.cc:1035] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:01.606695 18195 raft_consensus.cc:2932] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:01.607379 13447 raft_consensus.cc:2932] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:01.612919 13578 raft_consensus.cc:2932] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:01.613006 13036 catalog_manager.cc:5484] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:01.619117 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:01.624596 13447 raft_consensus.cc:1261] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:01.624706 16420 raft_consensus.cc:1261] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:01.625100 18104 consensus_queue.cc:1035] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.625433 17813 consensus_queue.cc:1035] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.630465 13042 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet c2b5c6a57c10481a8a351de00b1a6a76 (table rebalancer_dynamic_table [id=4b9bb97f97c7437abbfb54275bfd92ea]). Expected version 1 got 0
I20240502 14:15:01.631580 18208 tablet.cc:1722] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:01.632496 18208 tablet.cc:1620] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:01.638123 13578 consensus_queue.cc:237] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:01.641249 18209 tablet.cc:1722] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:01.641908 18209 tablet.cc:1620] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:01.644590 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=4b9bb97f97c7437abbfb54275bfd92ea] alter complete (version 1)
I20240502 14:15:01.644747 13036 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet c2b5c6a57c10481a8a351de00b1a6a76 (table rebalancer_dynamic_table [id=4b9bb97f97c7437abbfb54275bfd92ea]). Expected version 1 got 0
I20240502 14:15:01.648238 13036 catalog_manager.cc:5576] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303) does not have the latest schema for tablet c2b5c6a57c10481a8a351de00b1a6a76 (table rebalancer_dynamic_table [id=4b9bb97f97c7437abbfb54275bfd92ea]). Expected version 1 got 0
I20240502 14:15:01.651129 17570 consensus_queue.cc:1035] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.653609 13317 raft_consensus.cc:1261] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:01.653884 16022 raft_consensus.cc:1261] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:01.654537 17794 consensus_queue.cc:1035] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:01.656868 17939 consensus_queue.cc:1035] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:01.658126 13515 consensus_peers.cc:485] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b648703b8fc742ecbe1dfd2047d4e68d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:01.672798 17705 raft_consensus.cc:2932] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:01.675977 16022 raft_consensus.cc:2932] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:01.680471 18213 tablet.cc:1722] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:01.682610 13036 catalog_manager.cc:5576] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303) does not have the latest schema for tablet c2b5c6a57c10481a8a351de00b1a6a76 (table rebalancer_dynamic_table [id=4b9bb97f97c7437abbfb54275bfd92ea]). Expected version 1 got 0
I20240502 14:15:01.684675 13036 catalog_manager.cc:5484] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:01.688016 18213 tablet.cc:1620] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: MemRowSet was empty: no flush needed.
I20240502 14:15:01.689039 13318 raft_consensus.cc:2932] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:01.690946 18219 consensus_queue.cc:1035] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.718140 16154 consensus_queue.cc:237] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:01.720486 16420 raft_consensus.cc:1261] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:01.720852 16022 raft_consensus.cc:1261] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:01.720934 16088 consensus_peers.cc:485] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb27b3bb29754d7399b0314ada6b1ba7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:01.721060 18229 consensus_queue.cc:1035] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:01.721388 17753 consensus_queue.cc:1035] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:01.723979 18229 raft_consensus.cc:2932] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:01.724969 16420 raft_consensus.cc:2932] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:01.725278 16022 raft_consensus.cc:2932] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:01.728602 13040 catalog_manager.cc:5484] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:01.750463 13036 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:01.759053 18175 raft_consensus.cc:482] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:01.759124 18208 tablet.cc:1722] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:01.759191 18175 raft_consensus.cc:504] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:01.759671 18208 tablet.cc:1620] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:01.759811 18175 leader_election.cc:288] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:01.760936 18209 tablet.cc:1722] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:01.760919 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb770f077c84cb99985ef59253f133b" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:01.761368 18209 tablet.cc:1620] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:01.761807 18213 tablet.cc:1722] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:01.762099 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb770f077c84cb99985ef59253f133b" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:01.762181 18213 tablet.cc:1620] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: MemRowSet was empty: no flush needed.
I20240502 14:15:01.762346 16022 raft_consensus.cc:2379] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8b47d4390f8343a493d6c00123785968 in current term 1: Already voted for candidate 5297e816f782430d9a01220f0471a882 in this term.
I20240502 14:15:01.762742 16355 leader_election.cc:302] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b47d4390f8343a493d6c00123785968; no voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:01.763149 18175 raft_consensus.cc:3037] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:01.765534 18175 raft_consensus.cc:2726] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:01.772007 13040 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=4b9bb97f97c7437abbfb54275bfd92ea] alter complete (version 2)
I20240502 14:15:01.771934 13578 consensus_queue.cc:237] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:01.773857 16154 raft_consensus.cc:1261] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:01.774012 13447 raft_consensus.cc:1261] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:01.774461 17939 consensus_queue.cc:1035] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:01.774746 17705 consensus_queue.cc:1035] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:01.776746 17939 raft_consensus.cc:2932] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:01.777360 13447 raft_consensus.cc:2932] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:01.778427 16154 raft_consensus.cc:2932] T 9ad2eb265e0f469787eb1699e339e0ed P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:01.782358 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9ad2eb265e0f469787eb1699e339e0ed with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:01.783265 13036 catalog_manager.cc:5484] T 9ad2eb265e0f469787eb1699e339e0ed P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:01.788966 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad2eb265e0f469787eb1699e339e0ed with delete_type TABLET_DATA_TOMBSTONED (TS 20a967bad5504b1e9c6fb720ecdd1625 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37690
I20240502 14:15:01.789803 18251 tablet_replica.cc:330] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:15:01.790104 18251 raft_consensus.cc:2229] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:01.790596 18251 raft_consensus.cc:2258] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:01.791481 18251 ts_tablet_manager.cc:1875] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:01.797916 18251 ts_tablet_manager.cc:1888] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:01.798096 18251 log.cc:1196] T 9ad2eb265e0f469787eb1699e339e0ed P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/9ad2eb265e0f469787eb1699e339e0ed
I20240502 14:15:01.798911 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 9ad2eb265e0f469787eb1699e339e0ed (table rebalance_test_table_15 [id=557f9d603bda4eb385d26221f3612c33]) successfully deleted
I20240502 14:15:01.833874 18264 ts_tablet_manager.cc:902] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:01.835963 18264 tablet_copy_client.cc:308] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:15:01.836866 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 819e6e83e6064c539cfc73a28c43db5b from peer 7eb4ecb2379b44c0954f2658c4a04350 ({username='slave'} at 127.6.196.196:49035)
I20240502 14:15:01.837045 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 819e6e83e6064c539cfc73a28c43db5b from peer 7eb4ecb2379b44c0954f2658c4a04350 at {username='slave'} at 127.6.196.196:49035: session id = 7eb4ecb2379b44c0954f2658c4a04350-819e6e83e6064c539cfc73a28c43db5b
I20240502 14:15:01.838976 13339 tablet_copy_source_session.cc:205] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:01.840396 18264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 819e6e83e6064c539cfc73a28c43db5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:01.844348 18264 tablet_copy_client.cc:783] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:01.844610 18264 tablet_copy_client.cc:647] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:01.846259 18264 tablet_copy_client.cc:523] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:01.849848 18264 tablet_bootstrap.cc:492] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:01.856622 18212 consensus_queue.cc:1035] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.858800 18264 tablet_bootstrap.cc:492] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:01.859464 18264 tablet_bootstrap.cc:492] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap complete.
I20240502 14:15:01.859761 18264 ts_tablet_manager.cc:1367] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240502 14:15:01.861131 18264 raft_consensus.cc:348] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:01.861438 18264 raft_consensus.cc:729] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: LEARNER
I20240502 14:15:01.861737 18264 consensus_queue.cc:260] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:01.862829 18264 ts_tablet_manager.cc:1398] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
I20240502 14:15:01.863528 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-819e6e83e6064c539cfc73a28c43db5b received from {username='slave'} at 127.6.196.196:49035
I20240502 14:15:01.863680 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-819e6e83e6064c539cfc73a28c43db5b on tablet 819e6e83e6064c539cfc73a28c43db5b with peer 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:15:01.868129 18104 consensus_queue.cc:1035] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:01.875535 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:01.885573 18208 tablet.cc:1722] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:01.889941 18213 tablet.cc:1722] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:01.895042 18209 tablet.cc:1722] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:01.895917 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=4b9bb97f97c7437abbfb54275bfd92ea] alter complete (version 3)
I20240502 14:15:01.999276 13042 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:01.999536 13042 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:02.002633 13042 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet c2b5c6a57c10481a8a351de00b1a6a76
I20240502 14:15:02.003348 16265 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b5c6a57c10481a8a351de00b1a6a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:01 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:02.003360 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b5c6a57c10481a8a351de00b1a6a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:01 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:02.004014 18334 tablet_replica.cc:330] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:02.004109 18333 tablet_replica.cc:330] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:02.004215 18334 raft_consensus.cc:2229] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:02.004225 16400 tablet_service.cc:1514] Processing DeleteTablet for tablet c2b5c6a57c10481a8a351de00b1a6a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:01 UTC) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:02.004339 18333 raft_consensus.cc:2229] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:02.004405 18334 raft_consensus.cc:2258] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:02.004700 18335 tablet_replica.cc:330] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:02.004930 18335 raft_consensus.cc:2229] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:02.005017 18333 raft_consensus.cc:2258] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:02.005168 18335 raft_consensus.cc:2258] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:02.005946 18335 ts_tablet_manager.cc:1875] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:02.005946 18334 ts_tablet_manager.cc:1875] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:02.010318 18333 ts_tablet_manager.cc:1875] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:02.011755 18335 ts_tablet_manager.cc:1888] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:02.011868 18335 log.cc:1196] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/c2b5c6a57c10481a8a351de00b1a6a76
I20240502 14:15:02.012300 18335 ts_tablet_manager.cc:1909] T c2b5c6a57c10481a8a351de00b1a6a76 P 8b47d4390f8343a493d6c00123785968: Deleting consensus metadata
I20240502 14:15:02.013566 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet c2b5c6a57c10481a8a351de00b1a6a76 (table rebalancer_dynamic_table_new_name [id=4b9bb97f97c7437abbfb54275bfd92ea]) successfully deleted
I20240502 14:15:02.016492 18334 ts_tablet_manager.cc:1888] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:02.016594 18334 log.cc:1196] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/c2b5c6a57c10481a8a351de00b1a6a76
I20240502 14:15:02.017055 18334 ts_tablet_manager.cc:1909] T c2b5c6a57c10481a8a351de00b1a6a76 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:15:02.017609 18338 ts_tablet_manager.cc:902] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d: Initiating tablet copy from peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:02.017993 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet c2b5c6a57c10481a8a351de00b1a6a76 (table rebalancer_dynamic_table_new_name [id=4b9bb97f97c7437abbfb54275bfd92ea]) successfully deleted
I20240502 14:15:02.019654 18333 ts_tablet_manager.cc:1888] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:02.019850 18333 log.cc:1196] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/c2b5c6a57c10481a8a351de00b1a6a76
I20240502 14:15:02.020294 18333 ts_tablet_manager.cc:1909] T c2b5c6a57c10481a8a351de00b1a6a76 P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:15:02.021361 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet c2b5c6a57c10481a8a351de00b1a6a76 (table rebalancer_dynamic_table_new_name [id=4b9bb97f97c7437abbfb54275bfd92ea]) successfully deleted
I20240502 14:15:02.021937 18338 tablet_copy_client.cc:308] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.200:43857
I20240502 14:15:02.023006 16442 tablet_copy_service.cc:140] P 8b47d4390f8343a493d6c00123785968: Received BeginTabletCopySession request for tablet 1b34f2a22306459784a1ad7b81cd1683 from peer cd67493f6f0142f896010f2b8a39887d ({username='slave'} at 127.6.196.194:47545)
I20240502 14:15:02.023166 16442 tablet_copy_service.cc:161] P 8b47d4390f8343a493d6c00123785968: Beginning new tablet copy session on tablet 1b34f2a22306459784a1ad7b81cd1683 from peer cd67493f6f0142f896010f2b8a39887d at {username='slave'} at 127.6.196.194:47545: session id = cd67493f6f0142f896010f2b8a39887d-1b34f2a22306459784a1ad7b81cd1683
I20240502 14:15:02.025722 16442 tablet_copy_source_session.cc:205] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:02.026860 18338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b34f2a22306459784a1ad7b81cd1683. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.032315 18338 tablet_copy_client.cc:783] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:02.032518 18338 tablet_copy_client.cc:647] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:02.033900 18338 tablet_copy_client.cc:523] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:02.036329 18338 tablet_bootstrap.cc:492] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:02.044538 18338 tablet_bootstrap.cc:492] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:02.044924 18338 tablet_bootstrap.cc:492] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d: Bootstrap complete.
I20240502 14:15:02.045183 18338 ts_tablet_manager.cc:1367] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240502 14:15:02.045918 18338 raft_consensus.cc:348] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:02.046167 18338 raft_consensus.cc:729] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: LEARNER
I20240502 14:15:02.046392 18338 consensus_queue.cc:260] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:02.046969 18338 ts_tablet_manager.cc:1398] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:02.047735 16442 tablet_copy_service.cc:342] P 8b47d4390f8343a493d6c00123785968: Request end of tablet copy session cd67493f6f0142f896010f2b8a39887d-1b34f2a22306459784a1ad7b81cd1683 received from {username='slave'} at 127.6.196.194:47545
I20240502 14:15:02.047895 16442 tablet_copy_service.cc:433] P 8b47d4390f8343a493d6c00123785968: ending tablet copy session cd67493f6f0142f896010f2b8a39887d-1b34f2a22306459784a1ad7b81cd1683 on tablet 1b34f2a22306459784a1ad7b81cd1683 with peer cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:02.050626 18352 ts_tablet_manager.cc:902] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:02.054777 13578 consensus_queue.cc:237] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:02.057281 13447 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:02.057508 13186 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:15:02.057721 13515 consensus_peers.cc:485] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16c817b864c4425eaa85ac4b5aff59fa. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:02.057855 17951 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:02.058133 17705 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:02.058627 18352 tablet_copy_client.cc:308] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:15:02.059828 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet b648703b8fc742ecbe1dfd2047d4e68d from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:53097)
I20240502 14:15:02.060005 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet b648703b8fc742ecbe1dfd2047d4e68d from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:53097: session id = 8b47d4390f8343a493d6c00123785968-b648703b8fc742ecbe1dfd2047d4e68d
I20240502 14:15:02.061591 17951 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:02.062372 13601 tablet_copy_source_session.cc:205] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:02.062680 13447 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:02.066902 18352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b648703b8fc742ecbe1dfd2047d4e68d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.067570 13040 catalog_manager.cc:5484] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:02.071609 17790 consensus_queue.cc:1035] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:02.073330 18352 tablet_copy_client.cc:783] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:02.073588 18352 tablet_copy_client.cc:647] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:02.078433 17786 consensus_queue.cc:1035] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:02.087028 18352 tablet_copy_client.cc:523] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:02.089679 13187 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:02.092011 18352 tablet_bootstrap.cc:492] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:02.098578 18362 ts_tablet_manager.cc:902] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:02.105644 18362 tablet_copy_client.cc:308] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:15:02.106410 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet 3fa51110016d4cafafbe25a1bd213f2e from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:40701)
I20240502 14:15:02.106578 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet 3fa51110016d4cafafbe25a1bd213f2e from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:40701: session id = d1da7f11f0604f78934354a992bb0f55-3fa51110016d4cafafbe25a1bd213f2e
I20240502 14:15:02.112041 13042 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:02.113006 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:02.110497 13601 tablet_copy_source_session.cc:205] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:02.118414 18264 ts_tablet_manager.cc:902] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:02.119045 18362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa51110016d4cafafbe25a1bd213f2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.128893 18362 tablet_copy_client.cc:783] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:02.129150 18362 tablet_copy_client.cc:647] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:02.133486 15999 tablet_service.cc:1467] Processing CreateTablet for tablet aad7ab7e91064c93a74a72e5a0c4059a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fc86f5aaca194455833a8a86aa79bb46]), partition=
I20240502 14:15:02.133780 16134 tablet_service.cc:1467] Processing CreateTablet for tablet aad7ab7e91064c93a74a72e5a0c4059a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fc86f5aaca194455833a8a86aa79bb46]), partition=
I20240502 14:15:02.133989 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad7ab7e91064c93a74a72e5a0c4059a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.134160 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad7ab7e91064c93a74a72e5a0c4059a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.138145 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 861f9171cb654ec889ddf824f2bb6e92 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=24518149b1dc474687818a2a77d62dda]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.138531 16400 tablet_service.cc:1467] Processing CreateTablet for tablet aad7ab7e91064c93a74a72e5a0c4059a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fc86f5aaca194455833a8a86aa79bb46]), partition=
I20240502 14:15:02.138619 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861f9171cb654ec889ddf824f2bb6e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.139012 16399 tablet_service.cc:1467] Processing CreateTablet for tablet dbe34cac4e774d3c9f57bc1d64569175 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=24518149b1dc474687818a2a77d62dda]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.139809 16133 tablet_service.cc:1467] Processing CreateTablet for tablet 861f9171cb654ec889ddf824f2bb6e92 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=24518149b1dc474687818a2a77d62dda]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.140200 16133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861f9171cb654ec889ddf824f2bb6e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.142539 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 861f9171cb654ec889ddf824f2bb6e92 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=24518149b1dc474687818a2a77d62dda]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.142910 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861f9171cb654ec889ddf824f2bb6e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.145570 16135 tablet_service.cc:1467] Processing CreateTablet for tablet e7b93f8449ef4e6fb0a7e315130da024 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=24518149b1dc474687818a2a77d62dda]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.145884 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b93f8449ef4e6fb0a7e315130da024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.157039 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe34cac4e774d3c9f57bc1d64569175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.161113 18375 tablet_bootstrap.cc:492] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:02.162003 18371 tablet_bootstrap.cc:492] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:02.162032 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aad7ab7e91064c93a74a72e5a0c4059a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.164028 18371 tablet_bootstrap.cc:654] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.174219 18352 tablet_bootstrap.cc:492] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:02.174707 18352 tablet_bootstrap.cc:492] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:15:02.175616 18375 tablet_bootstrap.cc:654] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.176735 18352 ts_tablet_manager.cc:1367] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.085s user 0.011s sys 0.011s
I20240502 14:15:02.177507 18352 raft_consensus.cc:348] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:02.177764 18352 raft_consensus.cc:729] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:15:02.178025 18352 consensus_queue.cc:260] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:02.182242 16265 tablet_service.cc:1467] Processing CreateTablet for tablet e7b93f8449ef4e6fb0a7e315130da024 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=24518149b1dc474687818a2a77d62dda]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.182693 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b93f8449ef4e6fb0a7e315130da024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.182816 16266 tablet_service.cc:1467] Processing CreateTablet for tablet dbe34cac4e774d3c9f57bc1d64569175 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=24518149b1dc474687818a2a77d62dda]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.183241 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe34cac4e774d3c9f57bc1d64569175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.186432 16002 tablet_service.cc:1467] Processing CreateTablet for tablet dbe34cac4e774d3c9f57bc1d64569175 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=24518149b1dc474687818a2a77d62dda]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.186887 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbe34cac4e774d3c9f57bc1d64569175. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.189050 18352 ts_tablet_manager.cc:1398] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:15:02.189779 13600 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-b648703b8fc742ecbe1dfd2047d4e68d received from {username='slave'} at 127.6.196.200:53097
I20240502 14:15:02.189913 13600 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session 8b47d4390f8343a493d6c00123785968-b648703b8fc742ecbe1dfd2047d4e68d on tablet b648703b8fc742ecbe1dfd2047d4e68d with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:15:02.197675 18381 ts_tablet_manager.cc:902] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:02.202719 18382 tablet_bootstrap.cc:492] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:02.203505 15999 tablet_service.cc:1467] Processing CreateTablet for tablet e7b93f8449ef4e6fb0a7e315130da024 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=24518149b1dc474687818a2a77d62dda]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:02.211050 18382 tablet_bootstrap.cc:654] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.211439 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b93f8449ef4e6fb0a7e315130da024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.225077 18382 tablet_bootstrap.cc:492] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:02.226125 18382 ts_tablet_manager.cc:1367] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20240502 14:15:02.229665 18382 raft_consensus.cc:348] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:02.231878 18382 raft_consensus.cc:374] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.232314 18382 raft_consensus.cc:729] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.232980 18382 consensus_queue.cc:260] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:02.237321 18382 ts_tablet_manager.cc:1398] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:15:02.242013 18382 tablet_bootstrap.cc:492] T dbe34cac4e774d3c9f57bc1d64569175 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:02.244046 18382 tablet_bootstrap.cc:654] T dbe34cac4e774d3c9f57bc1d64569175 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.258157 18362 tablet_copy_client.cc:523] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:02.258361 18379 ts_tablet_manager.cc:902] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:02.259546 18387 tablet_bootstrap.cc:492] T dbe34cac4e774d3c9f57bc1d64569175 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:02.261396 18387 tablet_bootstrap.cc:654] T dbe34cac4e774d3c9f57bc1d64569175 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.261868 18362 tablet_bootstrap.cc:492] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:02.262609 18388 tablet_bootstrap.cc:492] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:02.265012 18388 tablet_bootstrap.cc:654] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.268692 18381 tablet_copy_client.cc:308] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:36687
I20240502 14:15:02.269338 18375 tablet_bootstrap.cc:492] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:02.269618 18375 ts_tablet_manager.cc:1367] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.005s
I20240502 14:15:02.269593 13208 tablet_copy_service.cc:140] P 20a967bad5504b1e9c6fb720ecdd1625: Received BeginTabletCopySession request for tablet 2e51feadc94c48e9bcecffecacbda4b9 from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:48115)
I20240502 14:15:02.269766 13208 tablet_copy_service.cc:161] P 20a967bad5504b1e9c6fb720ecdd1625: Beginning new tablet copy session on tablet 2e51feadc94c48e9bcecffecacbda4b9 from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:48115: session id = 5297e816f782430d9a01220f0471a882-2e51feadc94c48e9bcecffecacbda4b9
I20240502 14:15:02.270208 18382 tablet_bootstrap.cc:492] T dbe34cac4e774d3c9f57bc1d64569175 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:02.270390 18382 ts_tablet_manager.cc:1367] T dbe34cac4e774d3c9f57bc1d64569175 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20240502 14:15:02.270737 18375 raft_consensus.cc:348] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:02.270992 18375 raft_consensus.cc:374] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.271112 18375 raft_consensus.cc:729] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.271327 18382 raft_consensus.cc:348] T dbe34cac4e774d3c9f57bc1d64569175 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.271538 18382 raft_consensus.cc:374] T dbe34cac4e774d3c9f57bc1d64569175 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.271628 18382 raft_consensus.cc:729] T dbe34cac4e774d3c9f57bc1d64569175 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.271852 18382 consensus_queue.cc:260] T dbe34cac4e774d3c9f57bc1d64569175 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.272444 18382 ts_tablet_manager.cc:1398] T dbe34cac4e774d3c9f57bc1d64569175 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:02.272599 13208 tablet_copy_source_session.cc:205] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:02.275944 18387 tablet_bootstrap.cc:492] T dbe34cac4e774d3c9f57bc1d64569175 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:02.276229 18387 ts_tablet_manager.cc:1367] T dbe34cac4e774d3c9f57bc1d64569175 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20240502 14:15:02.276293 18381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e51feadc94c48e9bcecffecacbda4b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.277215 18387 raft_consensus.cc:348] T dbe34cac4e774d3c9f57bc1d64569175 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.277482 18387 raft_consensus.cc:374] T dbe34cac4e774d3c9f57bc1d64569175 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.277595 18387 raft_consensus.cc:729] T dbe34cac4e774d3c9f57bc1d64569175 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.282178 18387 consensus_queue.cc:260] T dbe34cac4e774d3c9f57bc1d64569175 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.282815 18387 ts_tablet_manager.cc:1398] T dbe34cac4e774d3c9f57bc1d64569175 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:15:02.287237 18374 tablet_bootstrap.cc:492] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:02.287703 18381 tablet_copy_client.cc:783] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:02.287936 18381 tablet_copy_client.cc:647] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:02.288869 18374 tablet_bootstrap.cc:654] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.287506 18379 tablet_copy_client.cc:308] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.198:42041
I20240502 14:15:02.290107 18381 tablet_copy_client.cc:523] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:02.290536 16175 tablet_copy_service.cc:140] P c86f7440570f4796babfd44c3d7fd847: Received BeginTabletCopySession request for tablet eb27b3bb29754d7399b0314ada6b1ba7 from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:41633)
I20240502 14:15:02.290675 16175 tablet_copy_service.cc:161] P c86f7440570f4796babfd44c3d7fd847: Beginning new tablet copy session on tablet eb27b3bb29754d7399b0314ada6b1ba7 from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:41633: session id = 20a967bad5504b1e9c6fb720ecdd1625-eb27b3bb29754d7399b0314ada6b1ba7
I20240502 14:15:02.293695 18381 tablet_bootstrap.cc:492] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:02.293192 16175 tablet_copy_source_session.cc:205] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:02.296157 18379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb27b3bb29754d7399b0314ada6b1ba7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.297040 18375 consensus_queue.cc:260] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:02.301455 18375 ts_tablet_manager.cc:1398] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240502 14:15:02.304612 18374 tablet_bootstrap.cc:492] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:02.304960 18374 ts_tablet_manager.cc:1367] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.005s
I20240502 14:15:02.306363 18374 raft_consensus.cc:348] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:02.306818 18379 tablet_copy_client.cc:783] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:02.308106 18379 tablet_copy_client.cc:647] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:02.309347 18374 raft_consensus.cc:374] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.309722 18374 raft_consensus.cc:729] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.310580 18374 consensus_queue.cc:260] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:02.314438 18371 tablet_bootstrap.cc:492] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:02.314633 18371 ts_tablet_manager.cc:1367] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.153s user 0.004s sys 0.028s
I20240502 14:15:02.317652 18371 raft_consensus.cc:348] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.318032 18371 raft_consensus.cc:374] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.318490 18371 raft_consensus.cc:729] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.318802 18371 consensus_queue.cc:260] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.321785 18387 tablet_bootstrap.cc:492] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:02.324628 18371 ts_tablet_manager.cc:1398] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240502 14:15:02.324854 18264 tablet_copy_client.cc:308] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:15:02.325474 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet 7852539de07744caaf385620c5166861 from peer 7eb4ecb2379b44c0954f2658c4a04350 ({username='slave'} at 127.6.196.196:51943)
I20240502 14:15:02.325604 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet 7852539de07744caaf385620c5166861 from peer 7eb4ecb2379b44c0954f2658c4a04350 at {username='slave'} at 127.6.196.196:51943: session id = 7eb4ecb2379b44c0954f2658c4a04350-7852539de07744caaf385620c5166861
I20240502 14:15:02.328222 13470 tablet_copy_source_session.cc:205] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:02.330775 18371 tablet_bootstrap.cc:492] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:02.332692 18371 tablet_bootstrap.cc:654] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.333755 18264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7852539de07744caaf385620c5166861. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.334436 18387 tablet_bootstrap.cc:654] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.334681 18379 tablet_copy_client.cc:523] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:02.336035 18388 tablet_bootstrap.cc:492] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:02.336225 18388 ts_tablet_manager.cc:1367] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.074s user 0.006s sys 0.006s
I20240502 14:15:02.336809 18374 ts_tablet_manager.cc:1398] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.032s user 0.000s sys 0.002s
I20240502 14:15:02.338201 18388 raft_consensus.cc:348] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.338897 18379 tablet_bootstrap.cc:492] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:02.338929 18388 raft_consensus.cc:374] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.340005 18388 raft_consensus.cc:729] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.340344 18388 consensus_queue.cc:260] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.340384 16021 consensus_queue.cc:237] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:02.346025 18371 tablet_bootstrap.cc:492] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:02.346204 18371 ts_tablet_manager.cc:1367] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240502 14:15:02.345575 18388 ts_tablet_manager.cc:1398] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 14:15:02.347244 18371 raft_consensus.cc:348] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.347570 18371 raft_consensus.cc:374] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.347669 18371 raft_consensus.cc:729] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.347968 18371 consensus_queue.cc:260] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.348523 18371 ts_tablet_manager.cc:1398] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:02.349407 18371 tablet_bootstrap.cc:492] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:02.349829 18388 tablet_bootstrap.cc:492] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:02.350905 18371 tablet_bootstrap.cc:654] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.358718 18388 tablet_bootstrap.cc:654] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
W20240502 14:15:02.365834 15955 consensus_peers.cc:485] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e3da1778819497bb3bd56f9ba451046. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:02.366411 18387 tablet_bootstrap.cc:492] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:02.366638 18387 ts_tablet_manager.cc:1367] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.004s
I20240502 14:15:02.366838 16420 raft_consensus.cc:1261] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:02.367447 18220 consensus_queue.cc:1035] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:02.367744 18387 raft_consensus.cc:348] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.367962 18387 raft_consensus.cc:374] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.368098 18387 raft_consensus.cc:729] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.368407 18387 consensus_queue.cc:260] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.372422 18387 ts_tablet_manager.cc:1398] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:15:02.375291 18371 tablet_bootstrap.cc:492] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:02.375366 18379 tablet_bootstrap.cc:492] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:02.375424 18371 ts_tablet_manager.cc:1367] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.002s
I20240502 14:15:02.375749 18379 tablet_bootstrap.cc:492] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:15:02.375983 18379 ts_tablet_manager.cc:1367] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.037s user 0.010s sys 0.001s
I20240502 14:15:02.376019 13317 raft_consensus.cc:1261] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:02.376132 18371 raft_consensus.cc:348] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:02.376332 18371 raft_consensus.cc:374] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.376441 18371 raft_consensus.cc:729] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.376621 18371 consensus_queue.cc:260] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:02.376937 18379 raft_consensus.cc:348] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:02.377079 18371 ts_tablet_manager.cc:1398] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:02.377161 18379 raft_consensus.cc:729] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:15:02.377410 18379 consensus_queue.cc:260] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:02.378831 18379 ts_tablet_manager.cc:1398] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 14:15:02.379168 17570 consensus_queue.cc:1035] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:02.379419 16175 tablet_copy_service.cc:342] P c86f7440570f4796babfd44c3d7fd847: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-eb27b3bb29754d7399b0314ada6b1ba7 received from {username='slave'} at 127.6.196.193:41633
I20240502 14:15:02.379546 16175 tablet_copy_service.cc:433] P c86f7440570f4796babfd44c3d7fd847: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-eb27b3bb29754d7399b0314ada6b1ba7 on tablet eb27b3bb29754d7399b0314ada6b1ba7 with peer 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:15:02.381601 18220 raft_consensus.cc:2932] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:02.390218 13578 raft_consensus.cc:1203] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:02.391667 18362 tablet_bootstrap.cc:492] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:02.392036 18362 tablet_bootstrap.cc:492] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:15:02.390511 16420 raft_consensus.cc:2932] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:02.392261 18362 ts_tablet_manager.cc:1367] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.130s user 0.045s sys 0.006s
I20240502 14:15:02.393103 18362 raft_consensus.cc:348] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:02.393368 18362 raft_consensus.cc:729] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:15:02.393556 18362 consensus_queue.cc:260] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:02.394169 18362 ts_tablet_manager.cc:1398] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240502 14:15:02.394733 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-3fa51110016d4cafafbe25a1bd213f2e received from {username='slave'} at 127.6.196.197:40701
I20240502 14:15:02.394845 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-3fa51110016d4cafafbe25a1bd213f2e on tablet 3fa51110016d4cafafbe25a1bd213f2e with peer d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:02.403132 18381 tablet_bootstrap.cc:492] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:02.403630 18381 tablet_bootstrap.cc:492] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882: Bootstrap complete.
I20240502 14:15:02.403923 18381 ts_tablet_manager.cc:1367] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.110s user 0.043s sys 0.030s
I20240502 14:15:02.404718 18381 raft_consensus.cc:348] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:02.405032 18381 raft_consensus.cc:729] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: LEARNER
I20240502 14:15:02.405330 18381 consensus_queue.cc:260] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:02.405984 18381 ts_tablet_manager.cc:1398] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:15:02.406677 13208 tablet_copy_service.cc:342] P 20a967bad5504b1e9c6fb720ecdd1625: Request end of tablet copy session 5297e816f782430d9a01220f0471a882-2e51feadc94c48e9bcecffecacbda4b9 received from {username='slave'} at 127.6.196.199:48115
I20240502 14:15:02.406917 13208 tablet_copy_service.cc:433] P 20a967bad5504b1e9c6fb720ecdd1625: ending tablet copy session 5297e816f782430d9a01220f0471a882-2e51feadc94c48e9bcecffecacbda4b9 on tablet 2e51feadc94c48e9bcecffecacbda4b9 with peer 5297e816f782430d9a01220f0471a882
I20240502 14:15:02.407119 18264 tablet_copy_client.cc:783] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:02.407330 18264 tablet_copy_client.cc:647] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:02.415032 13036 catalog_manager.cc:5484] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:02.416194 18388 tablet_bootstrap.cc:492] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:02.416404 18388 ts_tablet_manager.cc:1367] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.067s user 0.000s sys 0.003s
I20240502 14:15:02.417377 18388 raft_consensus.cc:348] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:02.417726 18388 raft_consensus.cc:374] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.417840 18388 raft_consensus.cc:729] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.418143 18388 consensus_queue.cc:260] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:02.420040 18388 ts_tablet_manager.cc:1398] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:02.420480 18388 tablet_bootstrap.cc:492] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:02.425316 13318 raft_consensus.cc:1203] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:02.448079 18264 tablet_copy_client.cc:523] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:02.450634 18388 tablet_bootstrap.cc:654] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:02.461447 18220 raft_consensus.cc:482] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:02.461616 18220 raft_consensus.cc:504] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.462397 18220 leader_election.cc:288] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:02.463481 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe34cac4e774d3c9f57bc1d64569175" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:02.463742 16284 raft_consensus.cc:2454] T dbe34cac4e774d3c9f57bc1d64569175 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:02.464270 15957 leader_election.cc:302] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:02.464614 18220 raft_consensus.cc:2781] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:02.464804 18220 raft_consensus.cc:482] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:02.464949 18220 raft_consensus.cc:3037] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.471026 18220 raft_consensus.cc:504] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.471509 18419 ts_tablet_manager.cc:902] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:02.472209 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe34cac4e774d3c9f57bc1d64569175" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:02.472457 16284 raft_consensus.cc:3037] T dbe34cac4e774d3c9f57bc1d64569175 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.452189 18264 tablet_bootstrap.cc:492] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:02.477298 16284 raft_consensus.cc:2454] T dbe34cac4e774d3c9f57bc1d64569175 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:02.477792 15957 leader_election.cc:302] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:02.478127 17570 raft_consensus.cc:2781] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:02.478281 17570 raft_consensus.cc:686] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:02.478500 17570 consensus_queue.cc:237] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.481703 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe34cac4e774d3c9f57bc1d64569175" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:02.481997 16420 raft_consensus.cc:2454] T dbe34cac4e774d3c9f57bc1d64569175 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:02.483417 18220 leader_election.cc:288] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:02.483768 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbe34cac4e774d3c9f57bc1d64569175" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:02.484094 16420 raft_consensus.cc:3037] T dbe34cac4e774d3c9f57bc1d64569175 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.488430 16420 raft_consensus.cc:2454] T dbe34cac4e774d3c9f57bc1d64569175 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:02.486124 13040 catalog_manager.cc:5484] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:02.491468 13318 raft_consensus.cc:2932] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:02.496042 16420 raft_consensus.cc:1203] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:02.514016 16021 raft_consensus.cc:1203] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:02.521826 18388 tablet_bootstrap.cc:492] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:02.521998 18388 ts_tablet_manager.cc:1367] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.102s user 0.004s sys 0.000s
I20240502 14:15:02.522703 18388 raft_consensus.cc:348] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:02.522867 18388 raft_consensus.cc:374] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:02.522956 18388 raft_consensus.cc:729] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:02.523167 18388 consensus_queue.cc:260] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:02.523901 18388 ts_tablet_manager.cc:1398] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:02.554092 18220 raft_consensus.cc:482] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:02.554250 18220 raft_consensus.cc:504] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.554919 18220 leader_election.cc:288] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:02.555302 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad7ab7e91064c93a74a72e5a0c4059a" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:02.555567 16154 raft_consensus.cc:2454] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:02.556346 15956 leader_election.cc:302] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:02.557287 18220 raft_consensus.cc:2781] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:02.557418 18220 raft_consensus.cc:482] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:02.557518 18220 raft_consensus.cc:3037] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.560611 18220 raft_consensus.cc:504] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.561760 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad7ab7e91064c93a74a72e5a0c4059a" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:02.561954 16154 raft_consensus.cc:3037] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.562258 18229 raft_consensus.cc:482] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:02.562403 18229 raft_consensus.cc:504] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:02.563001 18229 leader_election.cc:288] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:02.563609 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b93f8449ef4e6fb0a7e315130da024" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:02.563829 16021 raft_consensus.cc:2454] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:02.564226 16090 leader_election.cc:302] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:02.564693 16154 raft_consensus.cc:2454] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:02.564749 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b93f8449ef4e6fb0a7e315130da024" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:02.564937 16284 raft_consensus.cc:2454] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:02.566717 17753 raft_consensus.cc:2781] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:02.566805 17753 raft_consensus.cc:482] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:02.566891 17753 raft_consensus.cc:3037] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.567171 15956 leader_election.cc:302] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:02.567472 17570 raft_consensus.cc:2781] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:02.567607 17570 raft_consensus.cc:686] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:02.567881 17570 consensus_queue.cc:237] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:02.569453 17753 raft_consensus.cc:504] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:02.570034 17753 leader_election.cc:288] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:02.570248 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad7ab7e91064c93a74a72e5a0c4059a" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:02.570472 16420 raft_consensus.cc:2454] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:02.571089 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b93f8449ef4e6fb0a7e315130da024" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:02.571287 16021 raft_consensus.cc:3037] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.573563 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7b93f8449ef4e6fb0a7e315130da024" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:02.573722 16284 raft_consensus.cc:3037] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.574342 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aad7ab7e91064c93a74a72e5a0c4059a" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:02.574514 16420 raft_consensus.cc:3037] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.574599 17939 raft_consensus.cc:482] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:02.574739 17939 raft_consensus.cc:504] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:02.575340 17939 leader_election.cc:288] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:02.576110 18220 leader_election.cc:288] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:02.576370 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861f9171cb654ec889ddf824f2bb6e92" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:02.576552 13186 raft_consensus.cc:2454] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:02.576625 16021 raft_consensus.cc:2454] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:02.576965 13514 leader_election.cc:302] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:15:02.577065 16090 leader_election.cc:302] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:02.577288 17939 raft_consensus.cc:2781] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:02.577283 16420 raft_consensus.cc:2454] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:02.577395 17939 raft_consensus.cc:482] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:02.577418 17753 raft_consensus.cc:2781] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:02.577488 17939 raft_consensus.cc:3037] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.577855 17753 raft_consensus.cc:686] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:02.578192 17753 consensus_queue.cc:237] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:02.579106 16284 raft_consensus.cc:2454] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:02.584976 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861f9171cb654ec889ddf824f2bb6e92" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:02.584650 17939 raft_consensus.cc:504] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:02.585464 16154 raft_consensus.cc:2454] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:02.586447 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861f9171cb654ec889ddf824f2bb6e92" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:02.586650 16152 raft_consensus.cc:3037] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.588387 18419 tablet_copy_client.cc:308] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:15:02.588958 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet 16c817b864c4425eaa85ac4b5aff59fa from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:42449)
I20240502 14:15:02.589107 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet 16c817b864c4425eaa85ac4b5aff59fa from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:42449: session id = c86f7440570f4796babfd44c3d7fd847-16c817b864c4425eaa85ac4b5aff59fa
I20240502 14:15:02.590938 16152 raft_consensus.cc:2454] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:02.591344 13515 leader_election.cc:302] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:02.591668 17951 raft_consensus.cc:2781] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:02.591818 17951 raft_consensus.cc:686] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:15:02.592105 13601 tablet_copy_source_session.cc:205] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:02.592072 17951 consensus_queue.cc:237] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:02.593290 18419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c817b864c4425eaa85ac4b5aff59fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:02.574167 13042 catalog_manager.cc:5484] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:02.601166 18419 tablet_copy_client.cc:783] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:02.601411 18419 tablet_copy_client.cc:647] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:02.602288 17939 leader_election.cc:288] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:02.602759 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "861f9171cb654ec889ddf824f2bb6e92" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:02.602972 13186 raft_consensus.cc:3037] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:02.603579 18419 tablet_copy_client.cc:523] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:02.603408 13042 catalog_manager.cc:5484] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:02.606964 18419 tablet_bootstrap.cc:492] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:02.610626 13186 raft_consensus.cc:2454] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:02.614639 13042 catalog_manager.cc:5484] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:02.625659 13445 consensus_queue.cc:237] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:02.628530 13578 raft_consensus.cc:1261] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:02.628799 13318 raft_consensus.cc:1261] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:02.629130 18167 consensus_queue.cc:1035] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:15:02.630600 13384 consensus_peers.cc:485] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dee785c61c73426fb5665c1334ff372b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:02.631458 18264 tablet_bootstrap.cc:492] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:02.631482 18167 consensus_queue.cc:1035] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:02.632009 18264 tablet_bootstrap.cc:492] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap complete.
I20240502 14:15:02.632298 18264 ts_tablet_manager.cc:1367] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.180s user 0.067s sys 0.005s
I20240502 14:15:02.633167 18264 raft_consensus.cc:348] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:02.633512 18264 raft_consensus.cc:729] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: LEARNER
I20240502 14:15:02.633755 18264 consensus_queue.cc:260] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:02.634476 18264 ts_tablet_manager.cc:1398] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:02.635861 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-7852539de07744caaf385620c5166861 received from {username='slave'} at 127.6.196.196:51943
I20240502 14:15:02.635998 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-7852539de07744caaf385620c5166861 on tablet 7852539de07744caaf385620c5166861 with peer 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:15:02.636286 17086 raft_consensus.cc:2932] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:02.637148 13318 raft_consensus.cc:2932] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:02.639097 13578 raft_consensus.cc:2932] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:02.643582 13042 catalog_manager.cc:5484] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:02.671641 13445 consensus_queue.cc:237] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:02.673830 13318 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:15:02.674012 13383 consensus_peers.cc:485] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e393a7736b14d5483973f9cb89058f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:02.674255 13578 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:02.674875 17086 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:02.675468 18438 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:02.677875 17086 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:02.678462 13318 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:02.678771 18419 tablet_bootstrap.cc:492] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:02.679250 18419 tablet_bootstrap.cc:492] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:15:02.679556 18419 ts_tablet_manager.cc:1367] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.073s user 0.032s sys 0.005s
I20240502 14:15:02.679478 13578 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:02.680359 18419 raft_consensus.cc:348] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:02.680598 18419 raft_consensus.cc:729] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:15:02.680800 18419 consensus_queue.cc:260] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:02.682281 18419 ts_tablet_manager.cc:1398] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20240502 14:15:02.683946 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-16c817b864c4425eaa85ac4b5aff59fa received from {username='slave'} at 127.6.196.198:42449
I20240502 14:15:02.684110 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-16c817b864c4425eaa85ac4b5aff59fa on tablet 16c817b864c4425eaa85ac4b5aff59fa with peer c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:02.684764 13040 catalog_manager.cc:5484] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:02.687256 16288 raft_consensus.cc:1203] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:02.708945 16154 consensus_queue.cc:237] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:02.711238 13318 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:15:02.711263 16089 consensus_peers.cc:485] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba8299c5f722440e92803100e7240723. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:02.711874 18457 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:02.712265 16021 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:02.712700 18457 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:02.714701 17753 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:02.715301 13318 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:02.715519 16022 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:02.719467 13040 catalog_manager.cc:5484] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 4 to 5, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:02.732900 13318 consensus_queue.cc:237] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:02.734791 16022 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:15:02.734863 13253 consensus_peers.cc:485] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a6b71f7e77b4084a8904b1045099472. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:02.735005 13578 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:02.735275 18414 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:02.735806 18414 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:02.737527 18413 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:02.738160 16022 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:02.738899 13578 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:02.743585 13040 catalog_manager.cc:5484] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 6 to 7, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:02.757750 13040 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:02.763239 16420 raft_consensus.cc:1261] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:02.763394 16154 raft_consensus.cc:1261] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:02.763703 17570 consensus_queue.cc:1035] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:02.764144 17570 consensus_queue.cc:1035] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:02.771982 16152 consensus_queue.cc:237] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:02.775806 16288 raft_consensus.cc:1261] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:02.776394 16090 consensus_peers.cc:485] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8e69d81920d147a38e2542d22ba8352a. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:02.777343 13186 raft_consensus.cc:1261] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:02.778255 17753 consensus_queue.cc:1035] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:02.778910 17753 consensus_queue.cc:1035] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:02.790163 13187 raft_consensus.cc:1203] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:02.792034 13040 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet aad7ab7e91064c93a74a72e5a0c4059a (table rebalancer_dynamic_table [id=fc86f5aaca194455833a8a86aa79bb46]). Expected version 1 got 0
I20240502 14:15:02.794732 17753 raft_consensus.cc:2932] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:02.798981 13188 raft_consensus.cc:2932] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:02.802148 18481 tablet.cc:1722] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:02.803653 18481 tablet.cc:1620] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:15:02.804566 16288 raft_consensus.cc:2932] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:02.808156 13040 catalog_manager.cc:5484] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:02.814640 13046 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=fc86f5aaca194455833a8a86aa79bb46] alter complete (version 1)
I20240502 14:15:02.832535 13040 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet aad7ab7e91064c93a74a72e5a0c4059a (table rebalancer_dynamic_table [id=fc86f5aaca194455833a8a86aa79bb46]). Expected version 1 got 0
I20240502 14:15:02.833070 13046 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet aad7ab7e91064c93a74a72e5a0c4059a (table rebalancer_dynamic_table [id=fc86f5aaca194455833a8a86aa79bb46]). Expected version 1 got 0
I20240502 14:15:02.834187 18483 tablet.cc:1722] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:02.835177 18483 tablet.cc:1620] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:02.840297 18486 tablet.cc:1722] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:02.841642 18486 tablet.cc:1620] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:02.843327 13040 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet aad7ab7e91064c93a74a72e5a0c4059a (table rebalancer_dynamic_table [id=fc86f5aaca194455833a8a86aa79bb46]). Expected version 1 got 0
I20240502 14:15:02.858244 13186 consensus_queue.cc:237] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:02.861299 13578 raft_consensus.cc:1261] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:02.861256 13445 raft_consensus.cc:1261] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:15:02.861974 13121 consensus_peers.cc:485] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a2fc48c889b481e9b4d249819a196a0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:02.862612 18198 consensus_queue.cc:1035] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:02.862944 18449 consensus_queue.cc:1035] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:02.879734 18198 raft_consensus.cc:2932] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:02.881948 13580 raft_consensus.cc:2932] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:02.888258 13040 catalog_manager.cc:5484] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:02.896589 17939 raft_consensus.cc:1050] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:15:02.897436 17939 consensus_queue.cc:237] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:02.900914 13447 raft_consensus.cc:2932] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:02.918936 13040 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:02.921186 16418 raft_consensus.cc:1261] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:02.921867 17951 consensus_queue.cc:1035] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:02.925751 16022 raft_consensus.cc:1261] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:02.925945 13318 raft_consensus.cc:1261] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:02.926892 17939 consensus_queue.cc:1035] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:02.927290 17951 consensus_queue.cc:1035] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:02.932515 17951 raft_consensus.cc:2932] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:02.936977 18481 tablet.cc:1722] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:02.937125 16022 raft_consensus.cc:2932] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:02.937542 18481 tablet.cc:1620] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:15:02.938804 16418 raft_consensus.cc:2932] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:02.940167 18483 tablet.cc:1722] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:02.942093 18483 tablet.cc:1620] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:02.943066 13318 raft_consensus.cc:2932] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:02.945252 13042 catalog_manager.cc:5484] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:02.968402 13046 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=fc86f5aaca194455833a8a86aa79bb46] alter complete (version 2)
I20240502 14:15:02.969765 18503 ts_tablet_manager.cc:902] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:02.979212 13580 consensus_queue.cc:237] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:02.982164 16021 raft_consensus.cc:1261] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:02.982375 16418 raft_consensus.cc:1261] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:02.982731 17939 raft_consensus.cc:1050] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER d1da7f11f0604f78934354a992bb0f55 to VOTER
I20240502 14:15:02.984064 18501 consensus_queue.cc:1035] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:02.985018 18220 consensus_queue.cc:1035] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:02.985663 16152 raft_consensus.cc:1203] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:02.988456 17951 consensus_queue.cc:1035] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:02.987918 17939 consensus_queue.cc:237] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:02.996412 18486 tablet.cc:1722] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:02.996924 18486 tablet.cc:1620] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:02.996946 18508 raft_consensus.cc:2932] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:02.997848 18175 raft_consensus.cc:1050] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968: attempting to promote NON_VOTER cd67493f6f0142f896010f2b8a39887d to VOTER
I20240502 14:15:02.997553 16021 raft_consensus.cc:2932] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:02.998827 18175 consensus_queue.cc:237] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:03.002075 18413 raft_consensus.cc:1050] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 to VOTER
I20240502 14:15:03.003149 18413 consensus_queue.cc:237] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:03.015434 17570 consensus_queue.cc:1035] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.015568 13046 catalog_manager.cc:5484] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 3 to 4, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:03.021036 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet b648703b8fc742ecbe1dfd2047d4e68d with delete_type TABLET_DATA_TOMBSTONED (TS cd67493f6f0142f896010f2b8a39887d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57492
I20240502 14:15:03.022794 13447 raft_consensus.cc:1261] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:03.023867 13186 raft_consensus.cc:1261] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:03.024868 18502 consensus_queue.cc:1035] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:03.027318 18502 consensus_queue.cc:1035] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:03.038744 16021 raft_consensus.cc:1261] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:03.039623 18514 consensus_queue.cc:1035] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:03.046136 13580 raft_consensus.cc:1261] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:03.046694 16022 raft_consensus.cc:1261] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:03.046985 18414 consensus_queue.cc:1035] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:03.047359 17940 consensus_queue.cc:1035] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:15:03.048540 13515 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.198:42041, user_credentials={real_user=slave}} blocked reactor thread for 53042.4us
I20240502 14:15:03.050238 16418 raft_consensus.cc:2932] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:03.059528 18520 tablet_replica.cc:330] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:15:03.060117 18503 tablet_copy_client.cc:308] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.197:43695
I20240502 14:15:03.061209 18520 raft_consensus.cc:2229] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:03.061359 16043 tablet_copy_service.cc:140] P d1da7f11f0604f78934354a992bb0f55: Received BeginTabletCopySession request for tablet 8e3da1778819497bb3bd56f9ba451046 from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:54191)
I20240502 14:15:03.061605 18520 raft_consensus.cc:2258] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:03.061584 16043 tablet_copy_service.cc:161] P d1da7f11f0604f78934354a992bb0f55: Beginning new tablet copy session on tablet 8e3da1778819497bb3bd56f9ba451046 from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:54191: session id = 20a967bad5504b1e9c6fb720ecdd1625-8e3da1778819497bb3bd56f9ba451046
I20240502 14:15:03.062603 18520 ts_tablet_manager.cc:1875] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:03.063022 18501 raft_consensus.cc:2932] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:03.063447 13186 raft_consensus.cc:2932] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:03.065140 16043 tablet_copy_source_session.cc:205] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:03.066263 13318 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:03.066267 16284 raft_consensus.cc:1261] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:03.067188 17790 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:03.068076 18414 consensus_queue.cc:1035] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:03.069631 18514 consensus_queue.cc:1035] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.072080 16154 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:03.073285 16285 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:03.073567 17790 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:03.073854 18503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3da1778819497bb3bd56f9ba451046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.073887 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:03.077090 17786 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:03.084861 17705 consensus_queue.cc:1035] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.090725 16021 raft_consensus.cc:2932] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:03.091352 18481 tablet.cc:1722] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:03.105032 18503 tablet_copy_client.cc:783] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:03.105317 18503 tablet_copy_client.cc:647] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:03.107540 18486 tablet.cc:1722] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:03.106932 18413 raft_consensus.cc:2932] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:03.110348 16288 raft_consensus.cc:2932] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:03.114068 13580 raft_consensus.cc:2932] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:03.122380 16022 raft_consensus.cc:2932] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:03.128741 13578 raft_consensus.cc:1203] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:15:03.128882 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=fc86f5aaca194455833a8a86aa79bb46] alter complete (version 3)
I20240502 14:15:03.129572 17453 raft_consensus.cc:1050] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625: attempting to promote NON_VOTER 5297e816f782430d9a01220f0471a882 to VOTER
I20240502 14:15:03.130463 17453 consensus_queue.cc:237] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:03.132753 17790 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:03.136070 13042 catalog_manager.cc:5484] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:03.137503 16155 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:03.138411 16285 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:03.143663 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b648703b8fc742ecbe1dfd2047d4e68d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:03.143353 13318 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:03.146862 13036 catalog_manager.cc:5484] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:03.151428 13040 catalog_manager.cc:5484] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:03.153069 18483 tablet.cc:1722] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:03.154884 16418 consensus_queue.cc:237] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:03.125561 13447 raft_consensus.cc:2932] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:03.157740 13318 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:03.158110 16285 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:03.158481 17790 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:03.158859 17786 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:03.170725 18229 consensus_queue.cc:1035] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.171134 13578 consensus_queue.cc:237] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:03.172646 18175 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:03.175216 16288 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:03.175483 13319 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:03.177772 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b34f2a22306459784a1ad7b81cd1683 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:03.179500 13318 consensus_queue.cc:237] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:03.180549 13045 catalog_manager.cc:5484] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:03.181401 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:03.184233 18503 tablet_copy_client.cc:523] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:03.185148 16284 raft_consensus.cc:1261] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:03.185688 17940 consensus_queue.cc:1035] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:03.186388 16020 raft_consensus.cc:1261] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:03.186705 18520 ts_tablet_manager.cc:1888] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:03.187548 13580 raft_consensus.cc:1261] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:03.187790 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b34f2a22306459784a1ad7b81cd1683 with delete_type TABLET_DATA_TOMBSTONED (TS c86f7440570f4796babfd44c3d7fd847 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:03.188143 18414 consensus_queue.cc:1035] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:03.188709 16287 raft_consensus.cc:2932] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:03.189607 13578 raft_consensus.cc:2932] T 819e6e83e6064c539cfc73a28c43db5b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:03.192595 18503 tablet_bootstrap.cc:492] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:03.194749 18502 consensus_queue.cc:1035] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:03.196403 18520 log.cc:1196] T b648703b8fc742ecbe1dfd2047d4e68d P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/b648703b8fc742ecbe1dfd2047d4e68d
I20240502 14:15:03.197458 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet b648703b8fc742ecbe1dfd2047d4e68d (table rebalancer_extra_table_10 [id=89586fac1cc04eb6b38593443f23b4c7]) successfully deleted
I20240502 14:15:03.199246 18513 raft_consensus.cc:2932] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:03.202126 16020 raft_consensus.cc:2932] T 3fa51110016d4cafafbe25a1bd213f2e P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:03.203858 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3fa51110016d4cafafbe25a1bd213f2e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:03.204414 13041 catalog_manager.cc:5484] T 819e6e83e6064c539cfc73a28c43db5b P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:03.210666 18503 tablet_bootstrap.cc:492] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:03.211278 18503 tablet_bootstrap.cc:492] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:15:03.211678 18503 ts_tablet_manager.cc:1367] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.007s
I20240502 14:15:03.212697 18503 raft_consensus.cc:348] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:03.213119 18503 raft_consensus.cc:729] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:15:03.213467 18503 consensus_queue.cc:260] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:03.214162 18503 ts_tablet_manager.cc:1398] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:03.215220 15999 tablet_service.cc:1467] Processing CreateTablet for tablet 244b205ab48f468a91bf54660ef4711a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ab60746d83874ba487a87d72e6092c84]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.215613 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244b205ab48f468a91bf54660ef4711a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.215881 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 244b205ab48f468a91bf54660ef4711a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ab60746d83874ba487a87d72e6092c84]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.216298 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244b205ab48f468a91bf54660ef4711a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.217603 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 244b205ab48f468a91bf54660ef4711a (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ab60746d83874ba487a87d72e6092c84]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.217718 13580 raft_consensus.cc:1261] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:03.218003 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 244b205ab48f468a91bf54660ef4711a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.218676 13427 tablet_service.cc:1467] Processing CreateTablet for tablet a110350f069e45869eea48967f77e5fc (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ab60746d83874ba487a87d72e6092c84]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.218770 17453 consensus_queue.cc:1035] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:03.219134 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a110350f069e45869eea48967f77e5fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.221858 16284 raft_consensus.cc:1261] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:03.222659 13449 raft_consensus.cc:1261] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:03.223241 18198 consensus_queue.cc:1035] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:03.223449 16266 tablet_service.cc:1467] Processing CreateTablet for tablet e95022a368ec47949d1003c39a7fe433 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ab60746d83874ba487a87d72e6092c84]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.223678 18198 consensus_queue.cc:1035] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:03.223855 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95022a368ec47949d1003c39a7fe433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.225762 16002 tablet_service.cc:1467] Processing CreateTablet for tablet e95022a368ec47949d1003c39a7fe433 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ab60746d83874ba487a87d72e6092c84]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.226132 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95022a368ec47949d1003c39a7fe433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.229904 16135 tablet_service.cc:1467] Processing CreateTablet for tablet e95022a368ec47949d1003c39a7fe433 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ab60746d83874ba487a87d72e6092c84]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.230315 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95022a368ec47949d1003c39a7fe433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.232450 18229 raft_consensus.cc:1050] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847: attempting to promote NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 to VOTER
I20240502 14:15:03.233191 18229 consensus_queue.cc:237] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:03.235157 16399 tablet_service.cc:1467] Processing CreateTablet for tablet a110350f069e45869eea48967f77e5fc (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ab60746d83874ba487a87d72e6092c84]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.235642 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a110350f069e45869eea48967f77e5fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.238914 16043 tablet_copy_service.cc:342] P d1da7f11f0604f78934354a992bb0f55: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-8e3da1778819497bb3bd56f9ba451046 received from {username='slave'} at 127.6.196.193:54191
I20240502 14:15:03.239097 16043 tablet_copy_service.cc:433] P d1da7f11f0604f78934354a992bb0f55: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-8e3da1778819497bb3bd56f9ba451046 on tablet 8e3da1778819497bb3bd56f9ba451046 with peer 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:15:03.242952 18545 ts_tablet_manager.cc:902] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:03.247467 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet 819e6e83e6064c539cfc73a28c43db5b with delete_type TABLET_DATA_TOMBSTONED (TS d1da7f11f0604f78934354a992bb0f55 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:03.247961 16265 tablet_service.cc:1467] Processing CreateTablet for tablet a110350f069e45869eea48967f77e5fc (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=ab60746d83874ba487a87d72e6092c84]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:03.248515 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a110350f069e45869eea48967f77e5fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.263823 18457 consensus_queue.cc:1035] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.285348 18198 raft_consensus.cc:2932] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:03.292431 18554 tablet_bootstrap.cc:492] T 244b205ab48f468a91bf54660ef4711a P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:03.293015 13449 raft_consensus.cc:2932] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:03.295122 13186 raft_consensus.cc:1261] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:03.295540 18457 consensus_queue.cc:1035] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:03.305306 18541 tablet_replica.cc:330] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:03.305547 18541 raft_consensus.cc:2229] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:03.305797 18541 raft_consensus.cc:2258] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:03.306327 18551 tablet_bootstrap.cc:492] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:03.308487 18551 tablet_bootstrap.cc:654] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.310369 13042 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:03.310659 13042 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:03.311069 18547 ts_tablet_manager.cc:902] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:03.313088 16021 raft_consensus.cc:1261] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:03.313807 18457 consensus_queue.cc:1035] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:03.314972 13046 catalog_manager.cc:5484] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:03.315846 18541 ts_tablet_manager.cc:1875] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:03.316125 13042 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet aad7ab7e91064c93a74a72e5a0c4059a
I20240502 14:15:03.316987 16400 tablet_service.cc:1514] Processing DeleteTablet for tablet aad7ab7e91064c93a74a72e5a0c4059a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:03 UTC) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:03.317620 13045 catalog_manager.cc:5484] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:03.317556 18414 raft_consensus.cc:2932] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:03.318140 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet aad7ab7e91064c93a74a72e5a0c4059a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:03 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:03.323300 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 819e6e83e6064c539cfc73a28c43db5b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:03.324753 16002 tablet_service.cc:1514] Processing DeleteTablet for tablet aad7ab7e91064c93a74a72e5a0c4059a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:03 UTC) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:03.328830 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fa51110016d4cafafbe25a1bd213f2e with delete_type TABLET_DATA_TOMBSTONED (TS 20a967bad5504b1e9c6fb720ecdd1625 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:37690
I20240502 14:15:03.339646 18539 ts_tablet_manager.cc:902] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d: Initiating tablet copy from peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:03.340747 18554 tablet_bootstrap.cc:654] T 244b205ab48f468a91bf54660ef4711a P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.343081 18560 tablet_bootstrap.cc:492] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:03.344911 18560 tablet_bootstrap.cc:654] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.346737 18562 tablet_bootstrap.cc:492] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:03.348513 18551 tablet_bootstrap.cc:492] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:03.348551 18562 tablet_bootstrap.cc:654] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.349645 18559 tablet_bootstrap.cc:492] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:03.349658 18551 ts_tablet_manager.cc:1367] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.000s
I20240502 14:15:03.351022 18554 tablet_bootstrap.cc:492] T 244b205ab48f468a91bf54660ef4711a P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:03.352226 18559 tablet_bootstrap.cc:654] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.354802 18561 tablet_bootstrap.cc:492] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:03.355959 18571 ts_tablet_manager.cc:902] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:03.357045 18561 tablet_bootstrap.cc:654] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.357117 18554 ts_tablet_manager.cc:1367] T 244b205ab48f468a91bf54660ef4711a P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.001s
I20240502 14:15:03.356896 18551 raft_consensus.cc:348] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:03.363517 18551 raft_consensus.cc:374] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.363889 18551 raft_consensus.cc:729] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.364801 13445 raft_consensus.cc:1261] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:03.365926 18518 consensus_queue.cc:1035] T 3fa51110016d4cafafbe25a1bd213f2e P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:03.366382 18551 consensus_queue.cc:260] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:03.365751 18554 raft_consensus.cc:348] T 244b205ab48f468a91bf54660ef4711a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:03.367568 13445 raft_consensus.cc:2932] T 3fa51110016d4cafafbe25a1bd213f2e P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:03.366823 18554 raft_consensus.cc:374] T 244b205ab48f468a91bf54660ef4711a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.377210 16418 raft_consensus.cc:1261] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:03.378154 18457 consensus_queue.cc:1035] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:03.380743 16284 raft_consensus.cc:2932] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:03.381398 17086 raft_consensus.cc:482] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:03.381517 17086 raft_consensus.cc:504] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:03.382185 17086 leader_election.cc:288] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:03.382596 13186 consensus_queue.cc:237] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:03.383555 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a110350f069e45869eea48967f77e5fc" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
W20240502 14:15:03.384121 13384 leader_election.cc:341] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:03.384447 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a110350f069e45869eea48967f77e5fc" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
W20240502 14:15:03.385139 13383 leader_election.cc:341] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:03.385293 13383 leader_election.cc:302] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:03.386016 17086 raft_consensus.cc:2726] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:03.386478 18557 tablet_bootstrap.cc:492] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:03.386662 13445 raft_consensus.cc:1261] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:03.387274 18198 consensus_queue.cc:1035] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:03.387965 18557 tablet_bootstrap.cc:654] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.389300 18449 raft_consensus.cc:2932] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:03.389906 13445 raft_consensus.cc:2932] T 2e51feadc94c48e9bcecffecacbda4b9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:03.394089 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2e51feadc94c48e9bcecffecacbda4b9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:03.395512 16285 consensus_queue.cc:237] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:03.397194 13040 catalog_manager.cc:5484] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 5 to 6, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:03.397938 18561 tablet_bootstrap.cc:492] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:03.398149 18561 ts_tablet_manager.cc:1367] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20240502 14:15:03.398999 18561 raft_consensus.cc:348] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:03.399269 18561 raft_consensus.cc:374] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.399394 18561 raft_consensus.cc:729] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.399739 18561 consensus_queue.cc:260] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:03.400413 18553 ts_tablet_manager.cc:902] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:03.401249 18561 ts_tablet_manager.cc:1398] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:03.402257 18560 tablet_bootstrap.cc:492] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:03.402429 18560 ts_tablet_manager.cc:1367] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.059s user 0.000s sys 0.003s
I20240502 14:15:03.403097 18560 raft_consensus.cc:348] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:03.403318 18560 raft_consensus.cc:374] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.403419 18560 raft_consensus.cc:729] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.403649 18560 consensus_queue.cc:260] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:03.405140 18560 ts_tablet_manager.cc:1398] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:15:03.405472 18560 tablet_bootstrap.cc:492] T a110350f069e45869eea48967f77e5fc P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:03.415009 18562 tablet_bootstrap.cc:492] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:03.415201 18562 ts_tablet_manager.cc:1367] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.003s
I20240502 14:15:03.415944 18547 tablet_copy_client.cc:308] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:15:03.415889 18562 raft_consensus.cc:348] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:03.416179 18562 raft_consensus.cc:374] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.416298 18562 raft_consensus.cc:729] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.416520 18562 consensus_queue.cc:260] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:03.416877 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet dee785c61c73426fb5665c1334ff372b from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:56181)
I20240502 14:15:03.417032 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet dee785c61c73426fb5665c1334ff372b from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:56181: session id = 5297e816f782430d9a01220f0471a882-dee785c61c73426fb5665c1334ff372b
I20240502 14:15:03.419636 13470 tablet_copy_source_session.cc:205] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:03.420418 18562 ts_tablet_manager.cc:1398] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:15:03.423332 16419 raft_consensus.cc:1261] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:03.424485 18426 consensus_queue.cc:1035] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:03.424604 13041 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:03.401530 18556 tablet_replica.cc:330] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
W20240502 14:15:03.427002 16219 consensus_peers.cc:485] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 -> Peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Couldn't send request to peer 7eb4ecb2379b44c0954f2658c4a04350. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5348ba5435c74f509ed77993191ef002. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:03.427361 18556 raft_consensus.cc:2229] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:03.427664 18556 raft_consensus.cc:2258] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:03.428082 16154 raft_consensus.cc:1261] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:03.428560 18561 tablet_bootstrap.cc:492] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:03.432842 18560 tablet_bootstrap.cc:654] T a110350f069e45869eea48967f77e5fc P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.434888 18561 tablet_bootstrap.cc:654] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.436412 13427 tablet_service.cc:1467] Processing CreateTablet for tablet da3022956b3c4ab5acdd4f3ffc0a6870 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6edd66b23ec946d086b3670bcf0e3386]), partition=
I20240502 14:15:03.436465 16266 tablet_service.cc:1467] Processing CreateTablet for tablet da3022956b3c4ab5acdd4f3ffc0a6870 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6edd66b23ec946d086b3670bcf0e3386]), partition=
I20240502 14:15:03.436831 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da3022956b3c4ab5acdd4f3ffc0a6870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.437083 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da3022956b3c4ab5acdd4f3ffc0a6870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.439031 18426 consensus_queue.cc:1035] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:03.446337 13186 raft_consensus.cc:1203] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:03.447618 18413 raft_consensus.cc:482] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:03.447744 18413 raft_consensus.cc:504] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:03.448458 18413 leader_election.cc:288] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:03.449752 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244b205ab48f468a91bf54660ef4711a" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:03.449990 16021 raft_consensus.cc:2454] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:03.450459 13253 leader_election.cc:302] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:03.437857 18556 ts_tablet_manager.cc:1875] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:03.447679 16000 tablet_service.cc:1467] Processing CreateTablet for tablet da3022956b3c4ab5acdd4f3ffc0a6870 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=6edd66b23ec946d086b3670bcf0e3386]), partition=
I20240502 14:15:03.451908 18537 raft_consensus.cc:2781] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:03.452003 18537 raft_consensus.cc:482] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:03.452102 18537 raft_consensus.cc:3037] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.452114 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da3022956b3c4ab5acdd4f3ffc0a6870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.455464 18537 raft_consensus.cc:504] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:03.456387 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244b205ab48f468a91bf54660ef4711a" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:03.456606 16021 raft_consensus.cc:3037] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.459069 16021 raft_consensus.cc:2454] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:03.459619 13253 leader_election.cc:302] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:03.460286 18571 tablet_copy_client.cc:308] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:15:03.461059 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 8a6b71f7e77b4084a8904b1045099472 from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:34141)
I20240502 14:15:03.461273 18547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dee785c61c73426fb5665c1334ff372b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.461244 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 8a6b71f7e77b4084a8904b1045099472 from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:34141: session id = 5297e816f782430d9a01220f0471a882-8a6b71f7e77b4084a8904b1045099472
I20240502 14:15:03.462455 18537 leader_election.cc:288] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:03.462607 18537 raft_consensus.cc:2781] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:03.463311 18537 raft_consensus.cc:686] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:03.463544 18537 consensus_queue.cc:237] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:03.463744 18556 ts_tablet_manager.cc:1888] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:03.463850 18556 log.cc:1196] T 819e6e83e6064c539cfc73a28c43db5b P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/819e6e83e6064c539cfc73a28c43db5b
I20240502 14:15:03.463838 13339 tablet_copy_source_session.cc:205] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:03.465025 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet 819e6e83e6064c539cfc73a28c43db5b (table rebalancer_extra_table_12 [id=ef46a85b1ca942a9ae1f3a3aad6af281]) successfully deleted
I20240502 14:15:03.465116 18554 raft_consensus.cc:729] T 244b205ab48f468a91bf54660ef4711a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.465337 18554 consensus_queue.cc:260] T 244b205ab48f468a91bf54660ef4711a P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:03.465781 18571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a6b71f7e77b4084a8904b1045099472. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.375469 18551 ts_tablet_manager.cc:1398] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240502 14:15:03.471906 18554 ts_tablet_manager.cc:1398] T 244b205ab48f468a91bf54660ef4711a P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.114s user 0.000s sys 0.001s
I20240502 14:15:03.475929 18551 tablet_bootstrap.cc:492] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:03.402137 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e51feadc94c48e9bcecffecacbda4b9 with delete_type TABLET_DATA_TOMBSTONED (TS 7eb4ecb2379b44c0954f2658c4a04350 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:58526
I20240502 14:15:03.451455 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244b205ab48f468a91bf54660ef4711a" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:03.477316 13578 raft_consensus.cc:2454] T 244b205ab48f468a91bf54660ef4711a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:03.460481 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "244b205ab48f468a91bf54660ef4711a" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:03.477593 13579 raft_consensus.cc:3037] T 244b205ab48f468a91bf54660ef4711a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.480309 13579 raft_consensus.cc:2454] T 244b205ab48f468a91bf54660ef4711a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:03.480624 18571 tablet_copy_client.cc:783] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:03.480883 18571 tablet_copy_client.cc:647] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:03.378335 13580 raft_consensus.cc:2932] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:03.483039 16419 raft_consensus.cc:2932] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:03.481647 18593 raft_consensus.cc:2932] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:03.489245 13042 catalog_manager.cc:5484] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:03.491281 18571 tablet_copy_client.cc:523] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:03.494235 18551 tablet_bootstrap.cc:654] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.497167 18571 tablet_bootstrap.cc:492] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:03.497150 18541 ts_tablet_manager.cc:1888] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:03.497300 18541 log.cc:1196] T 1b34f2a22306459784a1ad7b81cd1683 P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/1b34f2a22306459784a1ad7b81cd1683
I20240502 14:15:03.498191 18561 tablet_bootstrap.cc:492] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:03.498399 18561 ts_tablet_manager.cc:1367] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.003s
I20240502 14:15:03.499293 18561 raft_consensus.cc:348] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:03.504478 18598 raft_consensus.cc:482] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:03.508018 18561 raft_consensus.cc:374] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.508164 18561 raft_consensus.cc:729] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.508038 18598 raft_consensus.cc:504] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:03.508500 18561 consensus_queue.cc:260] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:03.509153 18598 leader_election.cc:288] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:03.509284 18561 ts_tablet_manager.cc:1398] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 14:15:03.509168 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95022a368ec47949d1003c39a7fe433" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:03.509503 16021 raft_consensus.cc:2454] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:03.509598 18561 tablet_bootstrap.cc:492] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:03.509948 16221 leader_election.cc:302] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:03.511127 18561 tablet_bootstrap.cc:654] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.512290 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 1b34f2a22306459784a1ad7b81cd1683 (table rebalancer_extra_table_14 [id=71087d0bec7e450da33bd69a7f555355]) successfully deleted
I20240502 14:15:03.515059 18541 tablet_replica.cc:330] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:03.515807 18541 raft_consensus.cc:2229] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:03.517853 18593 raft_consensus.cc:2781] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:03.518034 18593 raft_consensus.cc:482] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:03.518118 18541 raft_consensus.cc:2258] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:03.518177 18593 raft_consensus.cc:3037] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.518184 18569 tablet_replica.cc:330] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:15:03.518476 18569 raft_consensus.cc:2229] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:03.518843 18569 raft_consensus.cc:2258] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:03.520462 18556 tablet_replica.cc:330] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:03.520756 18556 raft_consensus.cc:2229] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:03.521275 18556 raft_consensus.cc:2258] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:03.521435 18568 ts_tablet_manager.cc:902] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:03.521490 18593 raft_consensus.cc:504] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:03.522230 18567 tablet_replica.cc:330] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:03.522354 18556 ts_tablet_manager.cc:1875] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:03.522436 18567 raft_consensus.cc:2229] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:03.522641 16021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95022a368ec47949d1003c39a7fe433" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:03.522758 18567 raft_consensus.cc:2258] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:03.522918 16021 raft_consensus.cc:3037] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.525254 18541 ts_tablet_manager.cc:1875] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:03.525303 16021 raft_consensus.cc:2454] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:03.525739 16221 leader_election.cc:302] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:03.526261 18593 leader_election.cc:288] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:03.527936 18593 raft_consensus.cc:2781] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:03.528666 18229 raft_consensus.cc:2932] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:03.529660 18596 tablet_replica.cc:330] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:15:03.529724 18547 tablet_copy_client.cc:783] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:03.529868 18596 raft_consensus.cc:2229] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:03.530025 18547 tablet_copy_client.cc:647] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:03.530094 18596 raft_consensus.cc:2258] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:03.534510 16420 raft_consensus.cc:2932] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:03.537250 18557 tablet_bootstrap.cc:492] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:03.537441 18557 ts_tablet_manager.cc:1367] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.151s user 0.002s sys 0.001s
I20240502 14:15:03.539153 18557 raft_consensus.cc:348] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:03.539381 18557 raft_consensus.cc:374] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.539498 18557 raft_consensus.cc:729] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.539855 18557 consensus_queue.cc:260] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:03.541610 18557 ts_tablet_manager.cc:1398] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:15:03.541587 13186 raft_consensus.cc:2932] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:03.541971 16020 raft_consensus.cc:2932] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:03.546258 18547 tablet_copy_client.cc:523] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:03.546305 18569 ts_tablet_manager.cc:1875] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:03.549319 18593 raft_consensus.cc:686] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:03.549641 18593 consensus_queue.cc:237] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:03.550344 18561 tablet_bootstrap.cc:492] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:03.550503 18561 ts_tablet_manager.cc:1367] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20240502 14:15:03.551226 18561 raft_consensus.cc:348] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:03.551440 18561 raft_consensus.cc:374] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.551589 18561 raft_consensus.cc:729] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.551846 18561 consensus_queue.cc:260] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:03.552580 18561 ts_tablet_manager.cc:1398] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:03.552774 13045 catalog_manager.cc:5484] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
W20240502 14:15:03.553849 16220 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.200:43857, user_credentials={real_user=slave}} blocked reactor thread for 66899.1us
I20240502 14:15:03.553973 18596 ts_tablet_manager.cc:1875] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:03.554129 16287 raft_consensus.cc:1261] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:03.556243 18547 tablet_bootstrap.cc:492] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:03.558058 17453 consensus_queue.cc:1035] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:03.563629 13040 catalog_manager.cc:5484] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:03.577447 16284 raft_consensus.cc:2932] T 2e51feadc94c48e9bcecffecacbda4b9 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:03.578303 18560 tablet_bootstrap.cc:492] T a110350f069e45869eea48967f77e5fc P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:03.578186 16153 consensus_queue.cc:237] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:03.578510 18560 ts_tablet_manager.cc:1367] T a110350f069e45869eea48967f77e5fc P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.173s user 0.007s sys 0.009s
I20240502 14:15:03.579290 18560 raft_consensus.cc:348] T a110350f069e45869eea48967f77e5fc P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:03.579531 18560 raft_consensus.cc:374] T a110350f069e45869eea48967f77e5fc P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.579617 18560 raft_consensus.cc:729] T a110350f069e45869eea48967f77e5fc P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.580267 18560 consensus_queue.cc:260] T a110350f069e45869eea48967f77e5fc P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:03.581182 13186 raft_consensus.cc:1261] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:03.581766 18560 ts_tablet_manager.cc:1398] T a110350f069e45869eea48967f77e5fc P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:03.582041 18560 tablet_bootstrap.cc:492] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:03.583351 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95022a368ec47949d1003c39a7fe433" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:03.583981 17753 consensus_queue.cc:1035] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20240502 14:15:03.586020 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95022a368ec47949d1003c39a7fe433" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:03.587286 17753 raft_consensus.cc:2932] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:03.588711 18539 tablet_copy_client.cc:308] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.198:42041
I20240502 14:15:03.589632 16175 tablet_copy_service.cc:140] P c86f7440570f4796babfd44c3d7fd847: Received BeginTabletCopySession request for tablet 8e69d81920d147a38e2542d22ba8352a from peer cd67493f6f0142f896010f2b8a39887d ({username='slave'} at 127.6.196.194:59023)
I20240502 14:15:03.589946 16175 tablet_copy_service.cc:161] P c86f7440570f4796babfd44c3d7fd847: Beginning new tablet copy session on tablet 8e69d81920d147a38e2542d22ba8352a from peer cd67493f6f0142f896010f2b8a39887d at {username='slave'} at 127.6.196.194:59023: session id = cd67493f6f0142f896010f2b8a39887d-8e69d81920d147a38e2542d22ba8352a
W20240502 14:15:03.590162 16220 leader_election.cc:341] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:15:03.590517 16220 leader_election.cc:341] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:03.590654 18508 raft_consensus.cc:1050] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER c86f7440570f4796babfd44c3d7fd847 to VOTER
I20240502 14:15:03.591259 13188 raft_consensus.cc:2932] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:03.591437 18508 consensus_queue.cc:237] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:03.593155 18551 tablet_bootstrap.cc:492] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:03.593335 18551 ts_tablet_manager.cc:1367] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.117s user 0.002s sys 0.004s
I20240502 14:15:03.594290 18551 raft_consensus.cc:348] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:03.594555 18551 raft_consensus.cc:374] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.594626 18551 raft_consensus.cc:729] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.594916 18551 consensus_queue.cc:260] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:03.599952 18560 tablet_bootstrap.cc:654] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:03.599704 13041 catalog_manager.cc:5484] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:03.603955 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet eb27b3bb29754d7399b0314ada6b1ba7 with delete_type TABLET_DATA_TOMBSTONED (TS d1da7f11f0604f78934354a992bb0f55 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:03.604548 18556 ts_tablet_manager.cc:1888] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:03.604665 18556 log.cc:1196] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/aad7ab7e91064c93a74a72e5a0c4059a
I20240502 14:15:03.608457 18556 ts_tablet_manager.cc:1909] T aad7ab7e91064c93a74a72e5a0c4059a P d1da7f11f0604f78934354a992bb0f55: Deleting consensus metadata
I20240502 14:15:03.609261 18551 ts_tablet_manager.cc:1398] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240502 14:15:03.610648 13188 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:03.611315 18518 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:03.613265 16154 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:03.613828 18518 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:03.616469 16175 tablet_copy_source_session.cc:205] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:03.618777 18559 tablet_bootstrap.cc:492] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:03.620206 17086 raft_consensus.cc:1050] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 to VOTER
I20240502 14:15:03.621089 17086 consensus_queue.cc:237] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:03.621671 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb27b3bb29754d7399b0314ada6b1ba7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:03.622969 13445 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:03.623175 18559 ts_tablet_manager.cc:1367] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.274s user 0.002s sys 0.020s
I20240502 14:15:03.625023 18539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e69d81920d147a38e2542d22ba8352a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.625625 18508 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:03.542562 18567 ts_tablet_manager.cc:1875] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:03.550017 17790 raft_consensus.cc:482] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:03.631852 17790 raft_consensus.cc:504] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:03.632540 17790 leader_election.cc:288] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:03.634101 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet aad7ab7e91064c93a74a72e5a0c4059a (table rebalancer_dynamic_table_new_name [id=fc86f5aaca194455833a8a86aa79bb46]) successfully deleted
I20240502 14:15:03.635140 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a110350f069e45869eea48967f77e5fc" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:03.635387 16287 raft_consensus.cc:2454] T a110350f069e45869eea48967f77e5fc P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:15:03.635725 18556 tablet_replica.cc:330] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:03.635936 18556 raft_consensus.cc:2229] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:03.636160 18556 raft_consensus.cc:2258] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:03.636111 16355 leader_election.cc:302] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:03.636391 17790 raft_consensus.cc:2781] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:03.636485 17790 raft_consensus.cc:482] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:03.636582 17790 raft_consensus.cc:3037] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.630244 18545 tablet_copy_client.cc:308] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:15:03.637948 18559 raft_consensus.cc:348] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:03.638185 18559 raft_consensus.cc:374] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.638288 18559 raft_consensus.cc:729] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.638540 18559 consensus_queue.cc:260] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:03.639384 18559 ts_tablet_manager.cc:1398] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:03.639602 17790 raft_consensus.cc:504] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:03.641151 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet 0e393a7736b14d5483973f9cb89058f8 from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:52085)
I20240502 14:15:03.641319 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet 0e393a7736b14d5483973f9cb89058f8 from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:52085: session id = c86f7440570f4796babfd44c3d7fd847-0e393a7736b14d5483973f9cb89058f8
I20240502 14:15:03.642030 17790 leader_election.cc:288] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:03.643342 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a110350f069e45869eea48967f77e5fc" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:03.643507 13449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a110350f069e45869eea48967f77e5fc" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:03.643579 13447 raft_consensus.cc:2454] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:15:03.645025 13470 tablet_copy_source_session.cc:205] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:03.647456 18545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e393a7736b14d5483973f9cb89058f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.651094 13580 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:03.651589 18438 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:03.614107 18568 tablet_copy_client.cc:308] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.198:42041
I20240502 14:15:03.609748 16418 raft_consensus.cc:1261] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:03.654139 17753 consensus_queue.cc:1035] T eb27b3bb29754d7399b0314ada6b1ba7 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:03.655920 16418 raft_consensus.cc:2932] T eb27b3bb29754d7399b0314ada6b1ba7 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:03.660668 16154 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:03.661252 18438 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:03.663064 16175 tablet_copy_service.cc:140] P c86f7440570f4796babfd44c3d7fd847: Received BeginTabletCopySession request for tablet ba8299c5f722440e92803100e7240723 from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:58917)
I20240502 14:15:03.663260 16175 tablet_copy_service.cc:161] P c86f7440570f4796babfd44c3d7fd847: Beginning new tablet copy session on tablet ba8299c5f722440e92803100e7240723 from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:58917: session id = 8b47d4390f8343a493d6c00123785968-ba8299c5f722440e92803100e7240723
I20240502 14:15:03.664942 18541 ts_tablet_manager.cc:1888] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:03.665096 18541 log.cc:1196] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/aad7ab7e91064c93a74a72e5a0c4059a
I20240502 14:15:03.667605 16175 tablet_copy_source_session.cc:205] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:03.669102 18568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8299c5f722440e92803100e7240723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.669692 18541 ts_tablet_manager.cc:1909] T aad7ab7e91064c93a74a72e5a0c4059a P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:15:03.669579 13449 consensus_queue.cc:237] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:03.672793 13188 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:03.673362 18438 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:03.630641 16152 raft_consensus.cc:2932] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:03.678714 18518 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:03.680279 18567 ts_tablet_manager.cc:1888] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:03.680387 18567 log.cc:1196] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/aad7ab7e91064c93a74a72e5a0c4059a
I20240502 14:15:03.680902 18567 ts_tablet_manager.cc:1909] T aad7ab7e91064c93a74a72e5a0c4059a P 8b47d4390f8343a493d6c00123785968: Deleting consensus metadata
I20240502 14:15:03.683710 18545 tablet_copy_client.cc:783] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:03.683789 16153 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:03.684301 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet aad7ab7e91064c93a74a72e5a0c4059a (table rebalancer_dynamic_table_new_name [id=fc86f5aaca194455833a8a86aa79bb46]) successfully deleted
I20240502 14:15:03.684597 18545 tablet_copy_client.cc:647] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:03.686643 16284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a110350f069e45869eea48967f77e5fc" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:03.686874 16284 raft_consensus.cc:3037] T a110350f069e45869eea48967f77e5fc P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.686978 18545 tablet_copy_client.cc:523] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:03.690740 13447 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:03.692773 13041 catalog_manager.cc:5484] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:03.694510 13186 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
W20240502 14:15:03.695853 13382 consensus_peers.cc:485] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0dd07adcbb1f4bd4bcb13bd089c89974. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:03.697032 18596 ts_tablet_manager.cc:1888] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:03.697207 18596 log.cc:1196] T 2e51feadc94c48e9bcecffecacbda4b9 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/2e51feadc94c48e9bcecffecacbda4b9
I20240502 14:15:03.700062 16155 raft_consensus.cc:1261] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:03.700754 18438 consensus_queue.cc:1035] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:03.702312 17753 raft_consensus.cc:482] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:03.703961 18569 ts_tablet_manager.cc:1888] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:03.704135 18569 log.cc:1196] T 3fa51110016d4cafafbe25a1bd213f2e P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/3fa51110016d4cafafbe25a1bd213f2e
I20240502 14:15:03.704890 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 3fa51110016d4cafafbe25a1bd213f2e (table rebalance_test_table_5 [id=c97a45b4ee8c48a38ebd0709da5770f7]) successfully deleted
I20240502 14:15:03.706468 17753 raft_consensus.cc:504] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:03.708796 16287 raft_consensus.cc:1261] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:03.708869 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95022a368ec47949d1003c39a7fe433" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:03.709154 16020 raft_consensus.cc:2379] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c86f7440570f4796babfd44c3d7fd847 in current term 1: Already voted for candidate 5297e816f782430d9a01220f0471a882 in this term.
I20240502 14:15:03.709404 18556 ts_tablet_manager.cc:1875] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:03.709388 17086 consensus_queue.cc:1035] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:03.709962 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e95022a368ec47949d1003c39a7fe433" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:03.716200 18438 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:03.720504 18539 tablet_copy_client.cc:783] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:03.724891 13188 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:03.725728 13045 catalog_manager.cc:5484] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 7 to 8, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:03.703408 13579 consensus_queue.cc:237] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:03.733023 13186 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:03.733469 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 2e51feadc94c48e9bcecffecacbda4b9 (table rebalance_test_table_11 [id=b7310fe1c01b4fe7b29328321c07e72d]) successfully deleted
I20240502 14:15:03.734769 13579 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:03.739215 18539 tablet_copy_client.cc:647] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:03.740242 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet aad7ab7e91064c93a74a72e5a0c4059a (table rebalancer_dynamic_table_new_name [id=fc86f5aaca194455833a8a86aa79bb46]) successfully deleted
I20240502 14:15:03.742290 18508 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:03.742749 13040 catalog_manager.cc:5484] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:03.744738 18518 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:03.747458 18560 tablet_bootstrap.cc:492] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:03.747673 18560 ts_tablet_manager.cc:1367] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.166s user 0.003s sys 0.009s
I20240502 14:15:03.748548 18560 raft_consensus.cc:348] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:03.748606 13187 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:03.750499 18560 raft_consensus.cc:374] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:03.750612 18560 raft_consensus.cc:729] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:03.750840 18560 consensus_queue.cc:260] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:03.751313 18560 ts_tablet_manager.cc:1398] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:15:03.748679 16284 raft_consensus.cc:2454] T a110350f069e45869eea48967f77e5fc P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:15:03.752032 16355 leader_election.cc:302] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters: 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:15:03.752925 17790 raft_consensus.cc:2781] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:03.753072 17790 raft_consensus.cc:686] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:15:03.752913 18438 raft_consensus.cc:2932] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:03.754247 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 16c817b864c4425eaa85ac4b5aff59fa with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:03.754429 17790 consensus_queue.cc:237] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:03.755329 13045 catalog_manager.cc:5484] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:03.755558 18556 ts_tablet_manager.cc:1888] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:03.755688 18556 log.cc:1196] T eb27b3bb29754d7399b0314ada6b1ba7 P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/eb27b3bb29754d7399b0314ada6b1ba7
I20240502 14:15:03.757697 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet eb27b3bb29754d7399b0314ada6b1ba7 (table rebalancer_extra_table_12 [id=ef46a85b1ca942a9ae1f3a3aad6af281]) successfully deleted
I20240502 14:15:03.761603 17753 leader_election.cc:288] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:03.761968 16090 leader_election.cc:302] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847; no voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:03.764672 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c817b864c4425eaa85ac4b5aff59fa with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:03.765789 18457 raft_consensus.cc:3037] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.768918 18457 raft_consensus.cc:2726] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:03.772260 18593 raft_consensus.cc:482] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:03.772222 13045 catalog_manager.cc:5484] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:03.772459 18593 raft_consensus.cc:504] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:03.773612 18593 leader_election.cc:288] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:03.774072 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da3022956b3c4ab5acdd4f3ffc0a6870" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:03.774329 16020 raft_consensus.cc:2454] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:03.774428 16152 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:03.774804 16221 leader_election.cc:302] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:03.774920 18539 tablet_copy_client.cc:523] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:03.775048 18545 tablet_bootstrap.cc:492] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:03.773548 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da3022956b3c4ab5acdd4f3ffc0a6870" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:03.775331 13445 raft_consensus.cc:2454] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:03.776271 18593 raft_consensus.cc:2781] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:03.776372 18593 raft_consensus.cc:482] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:03.776479 18593 raft_consensus.cc:3037] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.779070 18593 raft_consensus.cc:504] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:03.780479 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da3022956b3c4ab5acdd4f3ffc0a6870" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:03.780715 16020 raft_consensus.cc:3037] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.783296 18508 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:03.783794 16020 raft_consensus.cc:2454] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:03.784240 16221 leader_election.cc:302] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:03.784780 18597 raft_consensus.cc:2781] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:03.785007 16152 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:03.785710 18597 raft_consensus.cc:686] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:03.786289 18597 consensus_queue.cc:237] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:03.786729 16288 raft_consensus.cc:2932] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:03.790994 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da3022956b3c4ab5acdd4f3ffc0a6870" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:03.791744 13445 raft_consensus.cc:3037] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:03.792177 18593 leader_election.cc:288] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:03.794281 18539 tablet_bootstrap.cc:492] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:03.796430 13445 raft_consensus.cc:2454] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:03.806914 16153 raft_consensus.cc:2932] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:03.808074 18568 tablet_copy_client.cc:783] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:03.808195 16152 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:03.813480 13042 catalog_manager.cc:5484] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:03.816494 18568 tablet_copy_client.cc:647] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:03.821949 18568 tablet_copy_client.cc:523] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:03.823892 13045 catalog_manager.cc:5484] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:03.774358 18637 tablet_replica.cc:330] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:03.832942 18637 raft_consensus.cc:2229] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:03.833261 18637 raft_consensus.cc:2258] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:03.838788 18568 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:03.838747 18553 tablet_copy_client.cc:308] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:36687
I20240502 14:15:03.839421 13208 tablet_copy_service.cc:140] P 20a967bad5504b1e9c6fb720ecdd1625: Received BeginTabletCopySession request for tablet 0a2fc48c889b481e9b4d249819a196a0 from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:46779)
I20240502 14:15:03.839596 13208 tablet_copy_service.cc:161] P 20a967bad5504b1e9c6fb720ecdd1625: Beginning new tablet copy session on tablet 0a2fc48c889b481e9b4d249819a196a0 from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:46779: session id = c86f7440570f4796babfd44c3d7fd847-0a2fc48c889b481e9b4d249819a196a0
I20240502 14:15:03.841804 18637 ts_tablet_manager.cc:1875] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:03.842217 18539 tablet_bootstrap.cc:492] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:03.842448 13208 tablet_copy_source_session.cc:205] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:03.842623 18539 tablet_bootstrap.cc:492] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d: Bootstrap complete.
I20240502 14:15:03.842836 18539 ts_tablet_manager.cc:1367] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.049s user 0.009s sys 0.000s
I20240502 14:15:03.843530 18539 raft_consensus.cc:348] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:03.843784 18539 raft_consensus.cc:729] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: LEARNER
I20240502 14:15:03.844002 18539 consensus_queue.cc:260] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:03.847566 18539 ts_tablet_manager.cc:1398] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:15:03.848182 16175 tablet_copy_service.cc:342] P c86f7440570f4796babfd44c3d7fd847: Request end of tablet copy session cd67493f6f0142f896010f2b8a39887d-8e69d81920d147a38e2542d22ba8352a received from {username='slave'} at 127.6.196.194:59023
I20240502 14:15:03.848331 16175 tablet_copy_service.cc:433] P c86f7440570f4796babfd44c3d7fd847: ending tablet copy session cd67493f6f0142f896010f2b8a39887d-8e69d81920d147a38e2542d22ba8352a on tablet 8e69d81920d147a38e2542d22ba8352a with peer cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:03.854564 18553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a2fc48c889b481e9b4d249819a196a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:03.878636 18637 ts_tablet_manager.cc:1888] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:03.878795 18637 log.cc:1196] T 16c817b864c4425eaa85ac4b5aff59fa P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/16c817b864c4425eaa85ac4b5aff59fa
I20240502 14:15:03.879805 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 16c817b864c4425eaa85ac4b5aff59fa (table rebalance_test_table_7 [id=fe675f309a11456aaf1fc329cfdb1707]) successfully deleted
I20240502 14:15:03.901098 18648 ts_tablet_manager.cc:902] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350: Initiating tablet copy from peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:03.903857 18568 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:03.910382 18568 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:15:03.911211 18571 tablet_bootstrap.cc:492] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:03.911549 18568 ts_tablet_manager.cc:1367] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.073s user 0.008s sys 0.000s
I20240502 14:15:03.911625 18571 tablet_bootstrap.cc:492] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882: Bootstrap complete.
I20240502 14:15:03.911867 18571 ts_tablet_manager.cc:1367] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.415s user 0.040s sys 0.015s
I20240502 14:15:03.912581 18571 raft_consensus.cc:348] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:03.912865 18571 raft_consensus.cc:729] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: LEARNER
I20240502 14:15:03.913054 18571 consensus_queue.cc:260] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:03.916548 18571 ts_tablet_manager.cc:1398] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:15:03.917477 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 5297e816f782430d9a01220f0471a882-8a6b71f7e77b4084a8904b1045099472 received from {username='slave'} at 127.6.196.199:34141
I20240502 14:15:03.917615 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 5297e816f782430d9a01220f0471a882-8a6b71f7e77b4084a8904b1045099472 on tablet 8a6b71f7e77b4084a8904b1045099472 with peer 5297e816f782430d9a01220f0471a882
I20240502 14:15:03.918609 18553 tablet_copy_client.cc:783] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:03.918912 18553 tablet_copy_client.cc:647] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:03.919076 18568 raft_consensus.cc:348] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:03.920341 18568 raft_consensus.cc:729] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:15:03.921886 18553 tablet_copy_client.cc:523] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:03.922006 18568 consensus_queue.cc:260] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:03.929898 18553 tablet_bootstrap.cc:492] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:03.929798 18537 consensus_queue.cc:1035] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.930835 18568 ts_tablet_manager.cc:1398] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240502 14:15:03.935979 13045 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:03.936355 16175 tablet_copy_service.cc:342] P c86f7440570f4796babfd44c3d7fd847: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-ba8299c5f722440e92803100e7240723 received from {username='slave'} at 127.6.196.200:58917
I20240502 14:15:03.936473 16175 tablet_copy_service.cc:433] P c86f7440570f4796babfd44c3d7fd847: ending tablet copy session 8b47d4390f8343a493d6c00123785968-ba8299c5f722440e92803100e7240723 on tablet ba8299c5f722440e92803100e7240723 with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:15:03.948112 18597 consensus_queue.cc:1035] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.968976 18547 tablet_bootstrap.cc:492] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:03.972780 16020 raft_consensus.cc:1261] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:03.973415 18597 consensus_queue.cc:1035] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.979254 13445 raft_consensus.cc:1261] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:03.980417 18537 consensus_queue.cc:1035] T 244b205ab48f468a91bf54660ef4711a P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.982532 18593 consensus_queue.cc:1035] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.983423 18547 tablet_bootstrap.cc:492] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882: Bootstrap complete.
I20240502 14:15:03.984207 18547 ts_tablet_manager.cc:1367] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.428s user 0.049s sys 0.013s
I20240502 14:15:03.985548 18598 consensus_queue.cc:1035] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:03.986238 18547 raft_consensus.cc:348] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:03.986451 18547 raft_consensus.cc:729] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: LEARNER
I20240502 14:15:03.986651 18547 consensus_queue.cc:260] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:03.988418 18547 ts_tablet_manager.cc:1398] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:03.990413 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session 5297e816f782430d9a01220f0471a882-dee785c61c73426fb5665c1334ff372b received from {username='slave'} at 127.6.196.199:56181
I20240502 14:15:03.990597 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session 5297e816f782430d9a01220f0471a882-dee785c61c73426fb5665c1334ff372b on tablet dee785c61c73426fb5665c1334ff372b with peer 5297e816f782430d9a01220f0471a882
I20240502 14:15:04.013525 18663 tablet.cc:1722] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:04.013970 18663 tablet.cc:1620] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:15:04.026098 13317 raft_consensus.cc:1203] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:04.029202 16288 consensus_queue.cc:237] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:04.051676 13449 raft_consensus.cc:1261] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:04.052424 16221 consensus_peers.cc:485] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: caa4af5f41674aaa8587158f25e495fb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:04.052702 18598 consensus_queue.cc:1035] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:04.055166 16022 raft_consensus.cc:1261] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:04.055919 18598 consensus_queue.cc:1035] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:04.079614 13045 catalog_manager.cc:5576] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303) does not have the latest schema for tablet da3022956b3c4ab5acdd4f3ffc0a6870 (table rebalancer_dynamic_table [id=6edd66b23ec946d086b3670bcf0e3386]). Expected version 1 got 0
I20240502 14:15:04.090533 18648 tablet_copy_client.cc:308] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.199:33857
I20240502 14:15:04.093631 16308 tablet_copy_service.cc:140] P 5297e816f782430d9a01220f0471a882: Received BeginTabletCopySession request for tablet 5348ba5435c74f509ed77993191ef002 from peer 7eb4ecb2379b44c0954f2658c4a04350 ({username='slave'} at 127.6.196.196:51405)
I20240502 14:15:04.093869 16308 tablet_copy_service.cc:161] P 5297e816f782430d9a01220f0471a882: Beginning new tablet copy session on tablet 5348ba5435c74f509ed77993191ef002 from peer 7eb4ecb2379b44c0954f2658c4a04350 at {username='slave'} at 127.6.196.196:51405: session id = 7eb4ecb2379b44c0954f2658c4a04350-5348ba5435c74f509ed77993191ef002
I20240502 14:15:04.096854 13040 catalog_manager.cc:5576] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303) does not have the latest schema for tablet da3022956b3c4ab5acdd4f3ffc0a6870 (table rebalancer_dynamic_table [id=6edd66b23ec946d086b3670bcf0e3386]). Expected version 1 got 0
I20240502 14:15:04.097934 16308 tablet_copy_source_session.cc:205] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:04.099138 18673 tablet.cc:1722] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:04.099982 18648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5348ba5435c74f509ed77993191ef002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.100035 18604 raft_consensus.cc:2932] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:04.106243 13449 raft_consensus.cc:2932] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:04.107383 13042 catalog_manager.cc:5484] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:04.107558 18673 tablet.cc:1620] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: MemRowSet was empty: no flush needed.
I20240502 14:15:04.109323 18675 tablet.cc:1722] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:04.110023 18675 tablet.cc:1620] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:04.119753 16288 raft_consensus.cc:1203] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:15:04.128675 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=6edd66b23ec946d086b3670bcf0e3386] alter complete (version 1)
I20240502 14:15:04.149988 18648 tablet_copy_client.cc:783] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:04.151816 18648 tablet_copy_client.cc:647] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:04.155946 18648 tablet_copy_client.cc:523] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:04.155932 16020 raft_consensus.cc:2932] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:04.157903 16418 raft_consensus.cc:1203] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:15:04.160688 18500 raft_consensus.cc:1050] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55: attempting to promote NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 to VOTER
I20240502 14:15:04.161828 18500 consensus_queue.cc:237] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:04.166317 17790 consensus_queue.cc:1035] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:04.171936 18648 tablet_bootstrap.cc:492] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:04.196035 18553 tablet_bootstrap.cc:492] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:04.196453 18553 tablet_bootstrap.cc:492] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:15:04.196715 18553 ts_tablet_manager.cc:1367] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.267s user 0.047s sys 0.015s
I20240502 14:15:04.196794 13186 raft_consensus.cc:1261] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:04.197892 18674 consensus_queue.cc:1035] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:04.199460 13317 raft_consensus.cc:1261] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:04.200206 18668 consensus_queue.cc:1035] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:04.201277 18553 raft_consensus.cc:348] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:04.201501 18553 raft_consensus.cc:729] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:15:04.201691 18553 consensus_queue.cc:260] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:04.205547 18553 ts_tablet_manager.cc:1398] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240502 14:15:04.218118 18668 raft_consensus.cc:2932] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:04.222637 18689 ts_tablet_manager.cc:902] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:04.222837 13449 consensus_queue.cc:237] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:04.225584 18648 tablet_bootstrap.cc:492] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:04.225293 13187 raft_consensus.cc:2932] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:04.226085 18648 tablet_bootstrap.cc:492] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap complete.
I20240502 14:15:04.226329 18648 ts_tablet_manager.cc:1367] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.011s
I20240502 14:15:04.227272 18648 raft_consensus.cc:348] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:04.227483 18648 raft_consensus.cc:729] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: LEARNER
I20240502 14:15:04.227249 13045 catalog_manager.cc:5484] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:04.227635 18648 consensus_queue.cc:260] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:04.233072 16418 raft_consensus.cc:1261] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:04.234730 18677 consensus_queue.cc:1035] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:04.237308 13208 tablet_copy_service.cc:342] P 20a967bad5504b1e9c6fb720ecdd1625: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-0a2fc48c889b481e9b4d249819a196a0 received from {username='slave'} at 127.6.196.198:46779
I20240502 14:15:04.237437 13208 tablet_copy_service.cc:433] P 20a967bad5504b1e9c6fb720ecdd1625: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-0a2fc48c889b481e9b4d249819a196a0 on tablet 0a2fc48c889b481e9b4d249819a196a0 with peer c86f7440570f4796babfd44c3d7fd847
W20240502 14:15:04.245726 13383 consensus_peers.cc:485] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d39f694c8554638a3b10ebf7cd01350. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:04.245754 13317 raft_consensus.cc:1261] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:04.246292 18635 consensus_queue.cc:1035] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:04.251646 13186 raft_consensus.cc:1261] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:04.252163 18635 consensus_queue.cc:1035] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:04.252234 13040 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:04.256846 13318 raft_consensus.cc:2932] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:04.267728 18648 ts_tablet_manager.cc:1398] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20240502 14:15:04.275247 18167 raft_consensus.cc:2932] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:04.279659 18675 tablet.cc:1722] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:04.280202 18675 tablet.cc:1620] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:04.282111 13045 catalog_manager.cc:5484] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:04.285871 16022 consensus_queue.cc:237] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:04.287065 18663 tablet.cc:1722] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:04.287596 18663 tablet.cc:1620] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:15:04.289307 13187 raft_consensus.cc:1261] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:04.290849 13580 raft_consensus.cc:1203] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:04.292531 18673 tablet.cc:1722] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:04.293054 18673 tablet.cc:1620] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: MemRowSet was empty: no flush needed.
I20240502 14:15:04.297991 16288 raft_consensus.cc:1203] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:04.298897 18674 consensus_queue.cc:1035] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:04.305945 13045 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=6edd66b23ec946d086b3670bcf0e3386] alter complete (version 2)
I20240502 14:15:04.308370 18674 raft_consensus.cc:2932] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:04.309513 13447 raft_consensus.cc:3037] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.312986 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8e3da1778819497bb3bd56f9ba451046 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:04.313660 13187 raft_consensus.cc:2932] T 8e3da1778819497bb3bd56f9ba451046 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:04.315024 18175 consensus_queue.cc:1035] T a110350f069e45869eea48967f77e5fc P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:04.316851 13042 catalog_manager.cc:5484] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:04.325582 16400 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3da1778819497bb3bd56f9ba451046 with delete_type TABLET_DATA_TOMBSTONED (TS 8b47d4390f8343a493d6c00123785968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:04.329077 16308 tablet_copy_service.cc:342] P 5297e816f782430d9a01220f0471a882: Request end of tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-5348ba5435c74f509ed77993191ef002 received from {username='slave'} at 127.6.196.196:51405
I20240502 14:15:04.329308 16308 tablet_copy_service.cc:433] P 5297e816f782430d9a01220f0471a882: ending tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-5348ba5435c74f509ed77993191ef002 on tablet 5348ba5435c74f509ed77993191ef002 with peer 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:15:04.331671 13318 raft_consensus.cc:1261] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:04.334503 18677 consensus_queue.cc:1035] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:04.337134 13045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:04.338090 13318 raft_consensus.cc:2932] T 8e3da1778819497bb3bd56f9ba451046 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:04.284265 13317 raft_consensus.cc:2932] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:04.347826 16135 tablet_service.cc:1467] Processing CreateTablet for tablet c78403038b614e6db2afb617328f6411 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a2fc7359aefc4890a033bb4d687e45ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.348286 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78403038b614e6db2afb617328f6411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.349438 16152 raft_consensus.cc:1203] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:04.352627 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 7a740ff5e0964efcb1fd7dd54dc4094c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a2fc7359aefc4890a033bb4d687e45ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.353094 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a740ff5e0964efcb1fd7dd54dc4094c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.355170 16266 tablet_service.cc:1467] Processing CreateTablet for tablet c78403038b614e6db2afb617328f6411 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a2fc7359aefc4890a033bb4d687e45ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.355628 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78403038b614e6db2afb617328f6411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.357630 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 7a740ff5e0964efcb1fd7dd54dc4094c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a2fc7359aefc4890a033bb4d687e45ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.358047 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a740ff5e0964efcb1fd7dd54dc4094c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.359155 16401 tablet_service.cc:1467] Processing CreateTablet for tablet cd06e00fd93340099df5e020716033a9 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a2fc7359aefc4890a033bb4d687e45ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.359493 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd06e00fd93340099df5e020716033a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.361737 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 7a740ff5e0964efcb1fd7dd54dc4094c (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a2fc7359aefc4890a033bb4d687e45ff]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.361804 16133 tablet_service.cc:1467] Processing CreateTablet for tablet cd06e00fd93340099df5e020716033a9 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a2fc7359aefc4890a033bb4d687e45ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.362114 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a740ff5e0964efcb1fd7dd54dc4094c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.365670 16133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd06e00fd93340099df5e020716033a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.352627 16400 tablet_service.cc:1467] Processing CreateTablet for tablet c78403038b614e6db2afb617328f6411 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a2fc7359aefc4890a033bb4d687e45ff]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.367954 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78403038b614e6db2afb617328f6411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.381052 13297 tablet_service.cc:1467] Processing CreateTablet for tablet cd06e00fd93340099df5e020716033a9 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=a2fc7359aefc4890a033bb4d687e45ff]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:04.381582 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd06e00fd93340099df5e020716033a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.384217 18698 tablet_bootstrap.cc:492] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:04.385979 18698 tablet_bootstrap.cc:654] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.387465 18545 tablet_bootstrap.cc:492] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:04.387899 18545 tablet_bootstrap.cc:492] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:15:04.388204 18545 ts_tablet_manager.cc:1367] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.679s user 0.041s sys 0.010s
I20240502 14:15:04.388212 18699 tablet_bootstrap.cc:492] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:04.390345 18699 tablet_bootstrap.cc:654] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.399293 16419 raft_consensus.cc:2932] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:04.400275 13188 raft_consensus.cc:2932] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:04.408480 18700 tablet_bootstrap.cc:492] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:04.413019 18698 tablet_bootstrap.cc:492] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:04.414491 18700 tablet_bootstrap.cc:654] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.416256 18698 ts_tablet_manager.cc:1367] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.004s
I20240502 14:15:04.422441 13040 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:04.424408 18697 tablet_replica.cc:330] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:04.424677 18697 raft_consensus.cc:2229] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:04.425029 18697 raft_consensus.cc:2258] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:04.426039 18697 ts_tablet_manager.cc:1875] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:04.436090 18545 raft_consensus.cc:348] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:04.436362 18545 raft_consensus.cc:729] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:15:04.437772 18675 tablet.cc:1722] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:04.438742 18673 tablet.cc:1722] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:04.442855 18545 consensus_queue.cc:260] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:04.443734 18701 tablet_bootstrap.cc:492] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:04.444604 18663 tablet.cc:1722] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:04.446321 18701 tablet_bootstrap.cc:654] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.447551 18703 tablet_bootstrap.cc:492] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:04.449788 18703 tablet_bootstrap.cc:654] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.452284 18699 tablet_bootstrap.cc:492] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:04.453176 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=6edd66b23ec946d086b3670bcf0e3386] alter complete (version 3)
I20240502 14:15:04.453449 18699 ts_tablet_manager.cc:1367] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.005s
I20240502 14:15:04.456434 18698 raft_consensus.cc:348] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:04.457494 18698 raft_consensus.cc:374] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.459617 18698 raft_consensus.cc:729] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.460264 18698 consensus_queue.cc:260] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:04.454772 18699 raft_consensus.cc:348] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:04.462083 18699 raft_consensus.cc:374] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.462379 18699 raft_consensus.cc:729] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.461727 18701 tablet_bootstrap.cc:492] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:04.463496 18701 ts_tablet_manager.cc:1367] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 14:15:04.464046 18545 ts_tablet_manager.cc:1398] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.076s user 0.005s sys 0.000s
I20240502 14:15:04.464715 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-0e393a7736b14d5483973f9cb89058f8 received from {username='slave'} at 127.6.196.198:52085
I20240502 14:15:04.464864 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-0e393a7736b14d5483973f9cb89058f8 on tablet 0e393a7736b14d5483973f9cb89058f8 with peer c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:04.464355 18701 raft_consensus.cc:348] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:04.465075 18701 raft_consensus.cc:374] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.465171 18701 raft_consensus.cc:729] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.466632 18701 consensus_queue.cc:260] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:04.467242 18701 ts_tablet_manager.cc:1398] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:15:04.467412 18698 ts_tablet_manager.cc:1398] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.051s user 0.000s sys 0.002s
I20240502 14:15:04.468808 18703 tablet_bootstrap.cc:492] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:04.469007 18703 ts_tablet_manager.cc:1367] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240502 14:15:04.468925 18699 consensus_queue.cc:260] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:04.469892 18703 raft_consensus.cc:348] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:04.470160 18703 raft_consensus.cc:374] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.470304 18703 raft_consensus.cc:729] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.470587 18703 consensus_queue.cc:260] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:04.471221 18703 ts_tablet_manager.cc:1398] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:04.471824 18703 tablet_bootstrap.cc:492] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:04.472354 18689 tablet_copy_client.cc:308] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:15:04.473045 18700 tablet_bootstrap.cc:492] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:04.473181 18700 ts_tablet_manager.cc:1367] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.000s
I20240502 14:15:04.473348 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet 0dd07adcbb1f4bd4bcb13bd089c89974 from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:43459)
I20240502 14:15:04.473520 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet 0dd07adcbb1f4bd4bcb13bd089c89974 from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:43459: session id = 20a967bad5504b1e9c6fb720ecdd1625-0dd07adcbb1f4bd4bcb13bd089c89974
I20240502 14:15:04.473893 18700 raft_consensus.cc:348] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:04.474074 18700 raft_consensus.cc:374] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.474152 18700 raft_consensus.cc:729] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.476372 13470 tablet_copy_source_session.cc:205] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:04.479393 18689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dd07adcbb1f4bd4bcb13bd089c89974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.481734 18699 ts_tablet_manager.cc:1398] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.015s user 0.000s sys 0.001s
I20240502 14:15:04.484643 17753 raft_consensus.cc:1050] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847: attempting to promote NON_VOTER cd67493f6f0142f896010f2b8a39887d to VOTER
I20240502 14:15:04.484942 18700 consensus_queue.cc:260] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:04.485423 17753 consensus_queue.cc:237] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:04.486780 18703 tablet_bootstrap.cc:654] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.488819 18229 raft_consensus.cc:482] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:04.488956 18229 raft_consensus.cc:504] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:04.489539 18229 leader_election.cc:288] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:04.489682 18697 ts_tablet_manager.cc:1888] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:04.489850 18697 log.cc:1196] T 8e3da1778819497bb3bd56f9ba451046 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/8e3da1778819497bb3bd56f9ba451046
I20240502 14:15:04.490800 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a740ff5e0964efcb1fd7dd54dc4094c" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:04.491010 16022 raft_consensus.cc:2454] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:04.491686 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 8e3da1778819497bb3bd56f9ba451046 (table rebalancer_extra_table_16 [id=bd0be61e86064cbca7fde36c32139e6a]) successfully deleted
I20240502 14:15:04.491916 16090 leader_election.cc:302] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:04.492163 18229 raft_consensus.cc:2781] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:04.492249 18229 raft_consensus.cc:482] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:04.492344 18229 raft_consensus.cc:3037] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.485566 18700 ts_tablet_manager.cc:1398] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240502 14:15:04.493537 18700 tablet_bootstrap.cc:492] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:04.494841 18229 raft_consensus.cc:504] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:04.495118 18700 tablet_bootstrap.cc:654] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.496111 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a740ff5e0964efcb1fd7dd54dc4094c" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:04.496567 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a740ff5e0964efcb1fd7dd54dc4094c" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:04.496759 16022 raft_consensus.cc:3037] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:15:04.496870 16089 leader_election.cc:341] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:04.498239 18689 tablet_copy_client.cc:783] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:04.498528 18689 tablet_copy_client.cc:647] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:04.499490 16022 raft_consensus.cc:2454] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:04.499910 16090 leader_election.cc:302] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:04.500233 18457 raft_consensus.cc:2781] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:04.501415 18457 raft_consensus.cc:686] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:04.501645 18703 tablet_bootstrap.cc:492] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:04.501673 18457 consensus_queue.cc:237] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:04.501798 18703 ts_tablet_manager.cc:1367] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.002s
I20240502 14:15:04.503871 16288 raft_consensus.cc:1261] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:04.504354 18703 raft_consensus.cc:348] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:04.505584 13045 catalog_manager.cc:5484] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:04.506089 13317 raft_consensus.cc:1261] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:04.507994 18457 consensus_queue.cc:1035] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:04.508484 18457 consensus_queue.cc:1035] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:04.504640 18703 raft_consensus.cc:374] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.509328 18703 raft_consensus.cc:729] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.509685 18703 consensus_queue.cc:260] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:04.512029 18703 ts_tablet_manager.cc:1398] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240502 14:15:04.512384 18703 tablet_bootstrap.cc:492] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:04.514340 18703 tablet_bootstrap.cc:654] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.516494 18229 leader_election.cc:288] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:04.518105 13188 raft_consensus.cc:1261] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:04.518714 18689 tablet_copy_client.cc:523] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:04.518690 18229 consensus_queue.cc:1035] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:04.520028 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a740ff5e0964efcb1fd7dd54dc4094c" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
W20240502 14:15:04.521147 16089 leader_election.cc:341] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:04.522493 18689 tablet_bootstrap.cc:492] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:04.528060 18700 tablet_bootstrap.cc:492] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:04.528273 18700 ts_tablet_manager.cc:1367] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.003s
I20240502 14:15:04.528404 18703 tablet_bootstrap.cc:492] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:04.528586 18703 ts_tablet_manager.cc:1367] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.005s
I20240502 14:15:04.529116 18700 raft_consensus.cc:348] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:04.529521 18700 raft_consensus.cc:374] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.529619 18700 raft_consensus.cc:729] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.530009 18700 consensus_queue.cc:260] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:04.530431 18703 raft_consensus.cc:348] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:04.530668 18703 raft_consensus.cc:374] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.530692 18700 ts_tablet_manager.cc:1398] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:04.530836 18703 raft_consensus.cc:729] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.531147 18703 consensus_queue.cc:260] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:04.532543 18700 tablet_bootstrap.cc:492] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:04.536090 18703 ts_tablet_manager.cc:1398] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 14:15:04.538914 18700 tablet_bootstrap.cc:654] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.551740 18689 tablet_bootstrap.cc:492] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:04.552119 18689 tablet_bootstrap.cc:492] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:15:04.552371 18689 ts_tablet_manager.cc:1367] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.003s
I20240502 14:15:04.552361 17753 raft_consensus.cc:2932] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:04.553050 18689 raft_consensus.cc:348] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:04.553318 18689 raft_consensus.cc:729] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:15:04.553524 18689 consensus_queue.cc:260] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:04.555437 18689 ts_tablet_manager.cc:1398] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240502 14:15:04.555728 13045 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:04.555929 13045 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:04.559367 13045 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet da3022956b3c4ab5acdd4f3ffc0a6870
I20240502 14:15:04.560192 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet da3022956b3c4ab5acdd4f3ffc0a6870 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:04 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:04.562325 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet da3022956b3c4ab5acdd4f3ffc0a6870 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:04 UTC) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:04.564064 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet da3022956b3c4ab5acdd4f3ffc0a6870 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:04 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:04.564683 13187 raft_consensus.cc:2932] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:04.570672 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-0dd07adcbb1f4bd4bcb13bd089c89974 received from {username='slave'} at 127.6.196.193:43459
I20240502 14:15:04.570842 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-0dd07adcbb1f4bd4bcb13bd089c89974 on tablet 0dd07adcbb1f4bd4bcb13bd089c89974 with peer 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:15:04.574481 13045 catalog_manager.cc:5484] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:04.577626 13317 raft_consensus.cc:2932] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:04.582949 18728 tablet_replica.cc:330] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:04.583354 18728 raft_consensus.cc:2229] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:04.586123 18728 raft_consensus.cc:2258] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:04.587940 18728 ts_tablet_manager.cc:1875] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:04.590516 18414 raft_consensus.cc:1050] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 5297e816f782430d9a01220f0471a882 to VOTER
I20240502 14:15:04.591672 18414 consensus_queue.cc:237] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:04.595854 16288 raft_consensus.cc:2932] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:04.605262 16152 consensus_queue.cc:237] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:04.610781 18728 ts_tablet_manager.cc:1888] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:04.611145 16284 raft_consensus.cc:1261] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:04.612638 18728 log.cc:1196] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/da3022956b3c4ab5acdd4f3ffc0a6870
I20240502 14:15:04.613281 18728 ts_tablet_manager.cc:1909] T da3022956b3c4ab5acdd4f3ffc0a6870 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:15:04.615002 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet da3022956b3c4ab5acdd4f3ffc0a6870 (table rebalancer_dynamic_table_new_name [id=6edd66b23ec946d086b3670bcf0e3386]) successfully deleted
I20240502 14:15:04.621730 13187 raft_consensus.cc:1203] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:04.622696 17753 consensus_queue.cc:1035] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:04.625633 18717 raft_consensus.cc:2932] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:04.626245 16288 raft_consensus.cc:2932] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:04.627820 16152 raft_consensus.cc:1203] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:04.631304 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8e69d81920d147a38e2542d22ba8352a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:04.641144 13040 catalog_manager.cc:5484] T 8e69d81920d147a38e2542d22ba8352a P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:04.646989 18730 tablet_replica.cc:330] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:04.647331 18730 raft_consensus.cc:2229] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:04.647650 18730 raft_consensus.cc:2258] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:04.648917 18730 ts_tablet_manager.cc:1875] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:04.649521 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e69d81920d147a38e2542d22ba8352a with delete_type TABLET_DATA_TOMBSTONED (TS 20a967bad5504b1e9c6fb720ecdd1625 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37690
I20240502 14:15:04.651796 13319 raft_consensus.cc:1261] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:04.655997 18730 ts_tablet_manager.cc:1888] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:04.656157 18730 log.cc:1196] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/da3022956b3c4ab5acdd4f3ffc0a6870
I20240502 14:15:04.656661 18730 ts_tablet_manager.cc:1909] T da3022956b3c4ab5acdd4f3ffc0a6870 P d1da7f11f0604f78934354a992bb0f55: Deleting consensus metadata
I20240502 14:15:04.657629 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet da3022956b3c4ab5acdd4f3ffc0a6870 (table rebalancer_dynamic_table_new_name [id=6edd66b23ec946d086b3670bcf0e3386]) successfully deleted
I20240502 14:15:04.662969 18700 tablet_bootstrap.cc:492] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:04.663189 18700 ts_tablet_manager.cc:1367] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.131s user 0.005s sys 0.001s
I20240502 14:15:04.664044 18700 raft_consensus.cc:348] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:04.664287 18700 raft_consensus.cc:374] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.664759 18700 raft_consensus.cc:729] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.664781 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:04.665303 18700 consensus_queue.cc:260] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:04.665771 17753 consensus_queue.cc:1035] T 8e69d81920d147a38e2542d22ba8352a P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:04.666905 18700 ts_tablet_manager.cc:1398] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:04.667311 13319 raft_consensus.cc:2932] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:04.668332 18735 ts_tablet_manager.cc:902] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d: Initiating tablet copy from peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:04.669329 18731 tablet_replica.cc:330] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:04.669520 18731 raft_consensus.cc:2229] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:04.669883 18731 raft_consensus.cc:2258] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:04.670207 16022 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:04.670748 16288 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:04.671291 13580 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:04.672703 18413 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:04.674906 18537 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:04.675433 18537 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:04.692489 16401 tablet_service.cc:1467] Processing CreateTablet for tablet fd2f7636fde94bf0b1790dbcc14332d4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1dc9be11772e41678fea5d654ee4210f]), partition=
I20240502 14:15:04.692569 16266 tablet_service.cc:1467] Processing CreateTablet for tablet fd2f7636fde94bf0b1790dbcc14332d4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1dc9be11772e41678fea5d654ee4210f]), partition=
I20240502 14:15:04.692879 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2f7636fde94bf0b1790dbcc14332d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.692900 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2f7636fde94bf0b1790dbcc14332d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.695899 13166 tablet_service.cc:1467] Processing CreateTablet for tablet fd2f7636fde94bf0b1790dbcc14332d4 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=1dc9be11772e41678fea5d654ee4210f]), partition=
I20240502 14:15:04.696244 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd2f7636fde94bf0b1790dbcc14332d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.698506 18698 tablet_bootstrap.cc:492] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:04.699301 18703 tablet_bootstrap.cc:492] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:04.699950 18698 tablet_bootstrap.cc:654] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.700861 18703 tablet_bootstrap.cc:654] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.706902 18537 raft_consensus.cc:482] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:04.707054 18537 raft_consensus.cc:504] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:04.707696 18537 leader_election.cc:288] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:04.708638 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd06e00fd93340099df5e020716033a9" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:04.708957 16419 raft_consensus.cc:2454] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:04.709713 13252 leader_election.cc:302] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:04.710429 18413 raft_consensus.cc:2781] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:04.710631 18413 raft_consensus.cc:482] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:04.710876 18413 raft_consensus.cc:3037] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.715103 18703 tablet_bootstrap.cc:492] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:04.715360 18703 ts_tablet_manager.cc:1367] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20240502 14:15:04.715293 18413 raft_consensus.cc:504] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:04.716490 18703 raft_consensus.cc:348] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:04.716883 18703 raft_consensus.cc:374] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.717300 18413 leader_election.cc:288] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:04.717232 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd06e00fd93340099df5e020716033a9" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:04.717470 16419 raft_consensus.cc:3037] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.720927 18731 ts_tablet_manager.cc:1875] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:04.721146 16419 raft_consensus.cc:2454] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:04.721506 18703 raft_consensus.cc:729] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.721772 18703 consensus_queue.cc:260] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:04.722388 18703 ts_tablet_manager.cc:1398] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 14:15:04.724586 13252 leader_election.cc:302] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:04.724855 18413 raft_consensus.cc:2781] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:04.725342 18413 raft_consensus.cc:686] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:04.725620 18413 consensus_queue.cc:237] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:04.725704 18698 tablet_bootstrap.cc:492] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:04.734439 18698 ts_tablet_manager.cc:1367] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.008s
I20240502 14:15:04.735373 18698 raft_consensus.cc:348] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:04.736017 18698 raft_consensus.cc:374] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:04.736097 18698 raft_consensus.cc:729] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:04.736948 18698 consensus_queue.cc:260] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:04.737738 18698 ts_tablet_manager.cc:1398] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:04.755479 18744 tablet_bootstrap.cc:492] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:04.757071 18744 tablet_bootstrap.cc:654] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:04.757174 13040 catalog_manager.cc:5484] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:04.758608 18414 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:04.760664 18731 ts_tablet_manager.cc:1888] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:04.761785 16022 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:04.762691 16284 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:04.765627 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd06e00fd93340099df5e020716033a9" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:04.765847 16154 raft_consensus.cc:2454] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:04.767694 18736 tablet_replica.cc:330] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:15:04.767643 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd06e00fd93340099df5e020716033a9" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:04.767798 16155 raft_consensus.cc:3037] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.767904 18736 raft_consensus.cc:2229] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:04.768174 18736 raft_consensus.cc:2258] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:04.769938 18735 tablet_copy_client.cc:308] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.199:33857
I20240502 14:15:04.770624 16155 raft_consensus.cc:2454] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:04.772480 18545 ts_tablet_manager.cc:902] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:04.779397 13042 catalog_manager.cc:5484] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 7 to 8, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:04.780462 16308 tablet_copy_service.cc:140] P 5297e816f782430d9a01220f0471a882: Received BeginTabletCopySession request for tablet caa4af5f41674aaa8587158f25e495fb from peer cd67493f6f0142f896010f2b8a39887d ({username='slave'} at 127.6.196.194:56211)
I20240502 14:15:04.780634 16308 tablet_copy_service.cc:161] P 5297e816f782430d9a01220f0471a882: Beginning new tablet copy session on tablet caa4af5f41674aaa8587158f25e495fb from peer cd67493f6f0142f896010f2b8a39887d at {username='slave'} at 127.6.196.194:56211: session id = cd67493f6f0142f896010f2b8a39887d-caa4af5f41674aaa8587158f25e495fb
I20240502 14:15:04.783210 16308 tablet_copy_source_session.cc:205] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:04.784360 18735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa4af5f41674aaa8587158f25e495fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:04.786531 18719 raft_consensus.cc:1050] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:15:04.787348 18719 consensus_queue.cc:237] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:04.787984 18736 ts_tablet_manager.cc:1875] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:04.788159 18438 raft_consensus.cc:1050] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER 5297e816f782430d9a01220f0471a882 to VOTER
I20240502 14:15:04.788897 18438 consensus_queue.cc:237] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:04.790517 18735 tablet_copy_client.cc:783] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:04.790839 18735 tablet_copy_client.cc:647] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:04.793644 13580 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:04.797765 18736 ts_tablet_manager.cc:1888] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:04.798532 13319 consensus_queue.cc:237] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:04.800030 18735 tablet_copy_client.cc:523] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:04.800311 16284 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:04.801286 18737 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20240502 14:15:04.802157 18736 log.cc:1196] T 8e69d81920d147a38e2542d22ba8352a P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/8e69d81920d147a38e2542d22ba8352a
I20240502 14:15:04.803153 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 8e69d81920d147a38e2542d22ba8352a (table rebalancer_extra_table_8 [id=296e981cee4f48689b6fcf85f9ab9562]) successfully deleted
I20240502 14:15:04.804551 18735 tablet_bootstrap.cc:492] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:04.807420 16020 raft_consensus.cc:1261] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:04.808148 18414 consensus_queue.cc:1035] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:04.808539 16420 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:04.810446 17753 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:04.813346 18414 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:04.815626 16022 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:04.822072 16288 raft_consensus.cc:1261] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:04.823081 13319 raft_consensus.cc:1261] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:04.824528 17086 consensus_queue.cc:1035] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:04.825089 17086 consensus_queue.cc:1035] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:04.828716 13318 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:04.829005 13580 raft_consensus.cc:1261] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:04.829335 16022 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:04.829308 17753 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:04.829618 18167 consensus_queue.cc:1035] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:04.831243 17753 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:04.835628 18598 raft_consensus.cc:1050] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882: attempting to promote NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 to VOTER
I20240502 14:15:04.836443 18449 raft_consensus.cc:1050] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625: attempting to promote NON_VOTER c86f7440570f4796babfd44c3d7fd847 to VOTER
I20240502 14:15:04.836808 18598 consensus_queue.cc:237] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:04.837440 18449 consensus_queue.cc:237] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:04.838894 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a6b71f7e77b4084a8904b1045099472 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:04.841604 13042 catalog_manager.cc:5484] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:04.845829 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a6b71f7e77b4084a8904b1045099472 with delete_type TABLET_DATA_TOMBSTONED (TS 7eb4ecb2379b44c0954f2658c4a04350 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58526
I20240502 14:15:04.847740 18735 tablet_bootstrap.cc:492] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:04.850172 13579 raft_consensus.cc:1261] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:04.850747 17453 consensus_queue.cc:1035] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:04.857131 16150 raft_consensus.cc:1261] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:04.859688 17453 consensus_queue.cc:1035] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:04.864291 18735 tablet_bootstrap.cc:492] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d: Bootstrap complete.
I20240502 14:15:04.865048 18735 ts_tablet_manager.cc:1367] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.004s
I20240502 14:15:04.874630 18604 raft_consensus.cc:482] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:04.874835 18604 raft_consensus.cc:504] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:04.875574 18604 leader_election.cc:288] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:04.875983 18735 raft_consensus.cc:348] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:04.876191 18735 raft_consensus.cc:729] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: LEARNER
I20240502 14:15:04.876371 18735 consensus_queue.cc:260] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:04.876508 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78403038b614e6db2afb617328f6411" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:04.876740 16419 raft_consensus.cc:2454] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:04.876853 18735 ts_tablet_manager.cc:1398] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:15:04.877372 16220 leader_election.cc:302] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:04.877488 16308 tablet_copy_service.cc:342] P 5297e816f782430d9a01220f0471a882: Request end of tablet copy session cd67493f6f0142f896010f2b8a39887d-caa4af5f41674aaa8587158f25e495fb received from {username='slave'} at 127.6.196.194:56211
I20240502 14:15:04.877611 16308 tablet_copy_service.cc:433] P 5297e816f782430d9a01220f0471a882: ending tablet copy session cd67493f6f0142f896010f2b8a39887d-caa4af5f41674aaa8587158f25e495fb on tablet caa4af5f41674aaa8587158f25e495fb with peer cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:04.878047 16151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78403038b614e6db2afb617328f6411" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:04.878274 16151 raft_consensus.cc:2454] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:04.878942 18604 raft_consensus.cc:2781] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:04.879027 18604 raft_consensus.cc:482] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:04.879117 18604 raft_consensus.cc:3037] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.881942 18604 raft_consensus.cc:504] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:04.883107 16151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78403038b614e6db2afb617328f6411" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:04.883287 16151 raft_consensus.cc:3037] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.886914 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c78403038b614e6db2afb617328f6411" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:04.887154 16419 raft_consensus.cc:3037] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.891295 18604 leader_election.cc:288] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:04.891568 16284 raft_consensus.cc:2932] T 8a6b71f7e77b4084a8904b1045099472 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:04.892175 16151 raft_consensus.cc:2454] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:04.892547 16220 leader_election.cc:302] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:04.893505 16419 raft_consensus.cc:2454] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:04.894116 18604 raft_consensus.cc:2781] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:04.894376 18604 raft_consensus.cc:686] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:04.894616 18604 consensus_queue.cc:237] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:04.898797 13445 raft_consensus.cc:1261] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:04.899469 18449 consensus_queue.cc:1035] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 14:15:04.900187 16419 raft_consensus.cc:3037] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.903015 17790 raft_consensus.cc:482] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:04.903174 17790 raft_consensus.cc:504] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:04.903411 17753 consensus_queue.cc:1035] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:04.903846 17790 leader_election.cc:288] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:04.904434 16151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a740ff5e0964efcb1fd7dd54dc4094c" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:04.905105 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a740ff5e0964efcb1fd7dd54dc4094c" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:04.905359 16020 raft_consensus.cc:2454] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:15:04.905974 16355 leader_election.cc:302] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters: c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:04.906394 17790 raft_consensus.cc:2781] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:04.906474 17790 raft_consensus.cc:482] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:04.906575 17790 raft_consensus.cc:3037] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:04.909566 17790 raft_consensus.cc:504] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:04.912026 17790 leader_election.cc:288] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 2 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:04.912595 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a740ff5e0964efcb1fd7dd54dc4094c" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:04.912801 16020 raft_consensus.cc:3037] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:04.915259 16020 raft_consensus.cc:2454] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 2.
I20240502 14:15:04.915730 16355 leader_election.cc:302] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:04.916198 16151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a740ff5e0964efcb1fd7dd54dc4094c" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:04.916657 18765 tablet_replica.cc:330] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:15:04.916890 18765 raft_consensus.cc:2229] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:04.917124 18765 raft_consensus.cc:2258] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:04.918399 17790 raft_consensus.cc:2781] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:04.918673 17790 raft_consensus.cc:686] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 2 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:15:04.918943 17790 consensus_queue.cc:237] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:04.922609 16151 raft_consensus.cc:1261] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:04.923230 18597 consensus_queue.cc:1035] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:04.923955 17753 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:04.924701 16418 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:04.930352 13318 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:04.935164 13042 catalog_manager.cc:5484] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 5 to 6, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:04.940142 16418 raft_consensus.cc:1261] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:04.940697 18597 consensus_queue.cc:1035] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:04.943120 16155 consensus_queue.cc:237] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:04.946301 16421 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:04.946306 13318 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:04.946437 13580 raft_consensus.cc:1261] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:04.947923 17753 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:04.948370 18729 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:04.948371 18598 consensus_queue.cc:1035] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:04.950119 18729 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:04.954205 13318 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:04.957994 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba8299c5f722440e92803100e7240723 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:04.960762 18449 raft_consensus.cc:2932] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:04.962270 13040 catalog_manager.cc:5484] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 6 to 7, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:04.965265 13579 raft_consensus.cc:2932] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:04.968240 18167 raft_consensus.cc:2932] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:04.973709 13045 catalog_manager.cc:5484] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:04.974237 18598 raft_consensus.cc:482] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:04.974653 18598 raft_consensus.cc:504] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:04.975427 13044 catalog_manager.cc:5484] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:04.975746 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2f7636fde94bf0b1790dbcc14332d4" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
W20240502 14:15:04.976449 16219 leader_election.cc:341] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:04.976691 18598 leader_election.cc:288] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:04.977573 13579 raft_consensus.cc:2932] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:04.978312 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet ba8299c5f722440e92803100e7240723 with delete_type TABLET_DATA_TOMBSTONED (TS d1da7f11f0604f78934354a992bb0f55 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:04.978541 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2f7636fde94bf0b1790dbcc14332d4" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:04.978745 16421 raft_consensus.cc:2454] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:04.981492 16220 leader_election.cc:302] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters: 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:15:04.982096 18598 raft_consensus.cc:2781] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:04.982496 18598 raft_consensus.cc:482] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:04.982730 18598 raft_consensus.cc:3037] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.982306 16155 raft_consensus.cc:2932] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:04.983947 16284 raft_consensus.cc:2932] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:04.986459 18598 raft_consensus.cc:504] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:04.988564 16421 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:04.989590 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2f7636fde94bf0b1790dbcc14332d4" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:04.989898 16418 raft_consensus.cc:3037] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:04.993577 16418 raft_consensus.cc:2454] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:04.994351 16220 leader_election.cc:302] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:04.994617 18597 raft_consensus.cc:2781] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:05.001427 13319 raft_consensus.cc:2932] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:05.006775 18765 ts_tablet_manager.cc:1875] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:05.013923 16419 raft_consensus.cc:1226] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 2 LEADER]: Rejecting Update request from peer c86f7440570f4796babfd44c3d7fd847 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
W20240502 14:15:05.016726 13514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.195:46303, user_credentials={real_user=slave}} blocked reactor thread for 113752us
I20240502 14:15:05.018088 18731 log.cc:1196] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/da3022956b3c4ab5acdd4f3ffc0a6870
I20240502 14:15:05.018785 18731 ts_tablet_manager.cc:1909] T da3022956b3c4ab5acdd4f3ffc0a6870 P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:15:05.019606 17753 consensus_queue.cc:1046] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.019927 17753 consensus_queue.cc:1225] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Peer 8b47d4390f8343a493d6c00123785968 log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20240502 14:15:05.020082 17753 raft_consensus.cc:3032] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:05.021528 17753 raft_consensus.cc:729] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:05.021924 17753 consensus_queue.cc:260] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:05.022295 17753 raft_consensus.cc:3037] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:05.024885 18744 tablet_bootstrap.cc:492] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:05.025090 18744 ts_tablet_manager.cc:1367] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.270s user 0.007s sys 0.014s
I20240502 14:15:05.025940 18744 raft_consensus.cc:348] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:05.026213 18744 raft_consensus.cc:374] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.026389 18744 raft_consensus.cc:729] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.026757 18744 consensus_queue.cc:260] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:05.027467 18744 ts_tablet_manager.cc:1398] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:05.031809 18765 ts_tablet_manager.cc:1888] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:15:05.031971 18765 log.cc:1196] T 8a6b71f7e77b4084a8904b1045099472 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/8a6b71f7e77b4084a8904b1045099472
I20240502 14:15:05.032322 18604 raft_consensus.cc:2932] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:05.035755 13444 raft_consensus.cc:2932] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:05.041280 13046 catalog_manager.cc:5484] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 1 to 2, leader changed from c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 2 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:05.041395 13045 catalog_manager.cc:5484] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:05.043138 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 8a6b71f7e77b4084a8904b1045099472 (table rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]) successfully deleted
I20240502 14:15:05.045325 13187 consensus_queue.cc:237] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:05.047338 13580 raft_consensus.cc:1261] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:05.048005 18449 consensus_queue.cc:1035] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:05.052920 18449 raft_consensus.cc:2932] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:05.057323 16151 raft_consensus.cc:1261] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:05.057467 18730 tablet_replica.cc:330] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:05.057699 18730 raft_consensus.cc:2229] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:05.057893 18730 pending_rounds.cc:62] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55: Trying to abort 1 pending ops.
I20240502 14:15:05.058058 18730 pending_rounds.cc:68] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7023244512406798336 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ba8299c5f722440e92803100e7240723" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } } }
I20240502 14:15:05.058419 18730 raft_consensus.cc:2894] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240502 14:15:05.058547 18730 raft_consensus.cc:2866] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:15:05.058631 18730 raft_consensus.cc:2258] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:05.058218 13580 raft_consensus.cc:2932] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:05.060060 18597 raft_consensus.cc:686] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:05.060350 18597 consensus_queue.cc:237] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:05.062391 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0a2fc48c889b481e9b4d249819a196a0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240502 14:15:05.062887 16219 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.195:46303, user_credentials={real_user=slave}} blocked reactor thread for 59712.7us
I20240502 14:15:05.063166 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet da3022956b3c4ab5acdd4f3ffc0a6870 (table rebalancer_dynamic_table_new_name [id=6edd66b23ec946d086b3670bcf0e3386]) successfully deleted
I20240502 14:15:05.063093 17453 consensus_queue.cc:1035] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:05.064970 16151 raft_consensus.cc:2932] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:05.064105 18598 leader_election.cc:288] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:05.066227 18545 tablet_copy_client.cc:308] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:15:05.068955 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd2f7636fde94bf0b1790dbcc14332d4" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:05.069202 13187 raft_consensus.cc:3037] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.069401 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet 3d39f694c8554638a3b10ebf7cd01350 from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:52085)
I20240502 14:15:05.069531 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet 3d39f694c8554638a3b10ebf7cd01350 from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:52085: session id = c86f7440570f4796babfd44c3d7fd847-3d39f694c8554638a3b10ebf7cd01350
I20240502 14:15:05.069501 13042 catalog_manager.cc:5484] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:05.072055 13187 raft_consensus.cc:2454] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:05.072432 13470 tablet_copy_source_session.cc:205] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:05.072919 13579 raft_consensus.cc:2932] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
W20240502 14:15:05.073541 16220 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.200:43857, user_credentials={real_user=slave}} blocked reactor thread for 46825.5us
I20240502 14:15:05.075184 18730 ts_tablet_manager.cc:1875] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:05.078745 13044 catalog_manager.cc:5484] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:05.082315 16420 raft_consensus.cc:2932] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:05.082340 16150 raft_consensus.cc:2932] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:05.087464 13319 raft_consensus.cc:1203] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:05.091311 13046 catalog_manager.cc:5484] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:05.092249 16288 consensus_queue.cc:237] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:05.092536 18545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d39f694c8554638a3b10ebf7cd01350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.096006 13579 raft_consensus.cc:1261] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:05.096585 18785 consensus_queue.cc:1035] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:05.097406 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a2fc48c889b481e9b4d249819a196a0 with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:05.097836 18728 tablet_replica.cc:330] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:05.098863 18728 raft_consensus.cc:2229] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:05.099071 18728 raft_consensus.cc:2258] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:05.102046 18785 raft_consensus.cc:2932] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:05.102725 18728 ts_tablet_manager.cc:1875] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:05.105695 18730 ts_tablet_manager.cc:1888] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:15:05.105681 13579 raft_consensus.cc:2932] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:05.110785 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5348ba5435c74f509ed77993191ef002 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:05.111063 18730 log.cc:1196] T ba8299c5f722440e92803100e7240723 P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/ba8299c5f722440e92803100e7240723
I20240502 14:15:05.113667 13042 catalog_manager.cc:5484] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:05.113991 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet ba8299c5f722440e92803100e7240723 (table rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]) successfully deleted
I20240502 14:15:05.115021 18545 tablet_copy_client.cc:783] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:05.115463 18545 tablet_copy_client.cc:647] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:05.115711 13444 consensus_queue.cc:237] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:05.116537 18728 ts_tablet_manager.cc:1888] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:05.116696 18728 log.cc:1196] T 0a2fc48c889b481e9b4d249819a196a0 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/0a2fc48c889b481e9b4d249819a196a0
I20240502 14:15:05.117386 18545 tablet_copy_client.cc:523] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:05.117925 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 0a2fc48c889b481e9b4d249819a196a0 (table rebalance_test_table_3 [id=8fa288258a0f4231be6de034088616ab]) successfully deleted
I20240502 14:15:05.118547 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet 5348ba5435c74f509ed77993191ef002 with delete_type TABLET_DATA_TOMBSTONED (TS 8b47d4390f8343a493d6c00123785968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:05.118676 16151 raft_consensus.cc:1261] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:05.120116 13580 raft_consensus.cc:1261] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:05.120949 18438 consensus_queue.cc:1035] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:05.122992 18790 tablet_replica.cc:330] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:05.123310 18790 raft_consensus.cc:2229] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:05.123628 18790 raft_consensus.cc:2258] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:05.123628 18788 consensus_queue.cc:1035] T 5348ba5435c74f509ed77993191ef002 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:05.124606 18790 ts_tablet_manager.cc:1875] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:05.125831 18438 raft_consensus.cc:2932] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:05.129204 13580 raft_consensus.cc:2932] T dee785c61c73426fb5665c1334ff372b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:05.132333 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dee785c61c73426fb5665c1334ff372b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:05.134932 13046 catalog_manager.cc:5484] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:05.139397 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet dee785c61c73426fb5665c1334ff372b with delete_type TABLET_DATA_TOMBSTONED (TS cd67493f6f0142f896010f2b8a39887d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57492
I20240502 14:15:05.139330 16151 raft_consensus.cc:2932] T 5348ba5435c74f509ed77993191ef002 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:05.144019 16288 raft_consensus.cc:1261] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:05.144621 18438 consensus_queue.cc:1035] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:05.149765 17086 raft_consensus.cc:1050] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 to VOTER
I20240502 14:15:05.150514 17086 consensus_queue.cc:237] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:05.151072 18790 ts_tablet_manager.cc:1888] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:05.151171 18790 log.cc:1196] T 5348ba5435c74f509ed77993191ef002 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/5348ba5435c74f509ed77993191ef002
I20240502 14:15:05.151849 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 5348ba5435c74f509ed77993191ef002 (table rebalancer_extra_table_16 [id=bd0be61e86064cbca7fde36c32139e6a]) successfully deleted
I20240502 14:15:05.153627 18793 tablet_replica.cc:330] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:15:05.153837 18793 raft_consensus.cc:2229] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:05.154181 18793 raft_consensus.cc:2258] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:05.155452 18793 ts_tablet_manager.cc:1875] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:05.156792 13187 raft_consensus.cc:1261] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:05.157647 18438 consensus_queue.cc:1035] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:05.160342 16150 raft_consensus.cc:1261] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:05.161845 18793 ts_tablet_manager.cc:1888] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:05.162227 16284 raft_consensus.cc:1261] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:05.162745 18793 log.cc:1196] T dee785c61c73426fb5665c1334ff372b P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/dee785c61c73426fb5665c1334ff372b
I20240502 14:15:05.163120 16288 raft_consensus.cc:2932] T dee785c61c73426fb5665c1334ff372b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:05.163619 18798 consensus_queue.cc:1035] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:05.163913 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet dee785c61c73426fb5665c1334ff372b (table rebalance_test_table_7 [id=fe675f309a11456aaf1fc329cfdb1707]) successfully deleted
I20240502 14:15:05.176873 18438 consensus_queue.cc:1035] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:05.177944 18737 consensus_queue.cc:1035] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.180619 18167 raft_consensus.cc:1050] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER c86f7440570f4796babfd44c3d7fd847 to VOTER
I20240502 14:15:05.181376 18167 consensus_queue.cc:237] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:05.199095 18545 tablet_bootstrap.cc:492] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:05.208917 16151 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:05.209487 13579 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:05.210126 18167 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:05.210269 13319 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:05.211961 18438 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:05.212672 18438 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:05.216572 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:05.243316 16154 raft_consensus.cc:2932] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:05.243685 17086 raft_consensus.cc:2932] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:05.259819 18537 consensus_queue.cc:1035] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.275346 13188 raft_consensus.cc:2932] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:05.281997 13042 catalog_manager.cc:5484] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:05.300921 18798 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:05.308310 13188 raft_consensus.cc:1261] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:05.309713 18788 consensus_queue.cc:1035] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.312247 13444 consensus_queue.cc:237] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:05.320130 13044 catalog_manager.cc:5484] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:05.324860 13187 raft_consensus.cc:1261] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:05.325603 18814 consensus_queue.cc:1035] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:05.326974 13319 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:05.329360 18798 raft_consensus.cc:2932] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:05.329942 13186 raft_consensus.cc:2932] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:05.333977 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0dd07adcbb1f4bd4bcb13bd089c89974 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:05.337272 16420 raft_consensus.cc:1261] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:05.337946 18788 consensus_queue.cc:1035] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.342866 13046 catalog_manager.cc:5484] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:05.351650 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dd07adcbb1f4bd4bcb13bd089c89974 with delete_type TABLET_DATA_TOMBSTONED (TS 5297e816f782430d9a01220f0471a882 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:05.354929 18731 tablet_replica.cc:330] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:05.355178 18731 raft_consensus.cc:2229] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:05.355326 18731 pending_rounds.cc:62] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882: Trying to abort 1 pending ops.
I20240502 14:15:05.355394 18731 pending_rounds.cc:68] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023244513894666240 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "0dd07adcbb1f4bd4bcb13bd089c89974" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } } }
I20240502 14:15:05.355746 18731 raft_consensus.cc:2894] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 14:15:05.355870 18731 raft_consensus.cc:2866] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:15:05.355957 18731 raft_consensus.cc:2258] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:05.356278 13046 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet fd2f7636fde94bf0b1790dbcc14332d4 (table rebalancer_dynamic_table [id=1dc9be11772e41678fea5d654ee4210f]). Expected version 1 got 0
I20240502 14:15:05.357978 13580 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:05.361595 18731 ts_tablet_manager.cc:1875] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:05.308071 16155 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:05.370735 18731 ts_tablet_manager.cc:1888] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:05.370975 18731 log.cc:1196] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/0dd07adcbb1f4bd4bcb13bd089c89974
I20240502 14:15:05.374891 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 0dd07adcbb1f4bd4bcb13bd089c89974 (table rebalancer_extra_table_14 [id=71087d0bec7e450da33bd69a7f555355]) successfully deleted
I20240502 14:15:05.376101 18545 tablet_bootstrap.cc:492] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:05.376547 18545 tablet_bootstrap.cc:492] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:15:05.376802 18545 ts_tablet_manager.cc:1367] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.178s user 0.049s sys 0.017s
I20240502 14:15:05.381510 16151 raft_consensus.cc:1261] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:05.381559 18545 raft_consensus.cc:348] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:05.381774 18545 raft_consensus.cc:729] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:15:05.382014 18545 consensus_queue.cc:260] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:05.382083 18167 consensus_queue.cc:1035] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:05.383785 18545 ts_tablet_manager.cc:1398] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
I20240502 14:15:05.383471 16152 raft_consensus.cc:2932] T 0dd07adcbb1f4bd4bcb13bd089c89974 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:05.385003 16150 raft_consensus.cc:1261] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20240502 14:15:05.385037 13042 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet fd2f7636fde94bf0b1790dbcc14332d4 (table rebalancer_dynamic_table [id=1dc9be11772e41678fea5d654ee4210f]). Expected version 1 got 0
I20240502 14:15:05.385113 16150 pending_rounds.cc:76] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847: Aborting all ops after (but not including) 0
I20240502 14:15:05.385161 16150 pending_rounds.cc:97] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20240502 14:15:05.385236 16150 raft_consensus.cc:2866] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:05.385764 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-3d39f694c8554638a3b10ebf7cd01350 received from {username='slave'} at 127.6.196.198:52085
I20240502 14:15:05.385757 17790 consensus_queue.cc:1035] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.385915 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-3d39f694c8554638a3b10ebf7cd01350 on tablet 3d39f694c8554638a3b10ebf7cd01350 with peer c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:05.389356 16022 raft_consensus.cc:1261] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240502 14:15:05.390027 18175 consensus_queue.cc:1035] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.400485 18820 tablet.cc:1722] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:05.401155 18820 tablet.cc:1620] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:05.408319 13046 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=1dc9be11772e41678fea5d654ee4210f] alter complete (version 1)
I20240502 14:15:05.424471 18821 tablet.cc:1722] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:05.425169 18821 tablet.cc:1620] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:05.430604 13044 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet fd2f7636fde94bf0b1790dbcc14332d4 (table rebalancer_dynamic_table [id=1dc9be11772e41678fea5d654ee4210f]). Expected version 1 got 0
I20240502 14:15:05.470072 18828 tablet.cc:1722] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:05.471983 18828 tablet.cc:1620] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:15:05.472527 13044 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet fd2f7636fde94bf0b1790dbcc14332d4 (table rebalancer_dynamic_table [id=1dc9be11772e41678fea5d654ee4210f]). Expected version 1 got 0
I20240502 14:15:05.514276 18788 consensus_queue.cc:1035] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.519449 18788 consensus_queue.cc:1035] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:05.523747 13046 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:05.530598 18783 raft_consensus.cc:1050] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882: attempting to promote NON_VOTER cd67493f6f0142f896010f2b8a39887d to VOTER
I20240502 14:15:05.531456 18783 consensus_queue.cc:237] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:05.533990 18820 tablet.cc:1722] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:05.534428 18820 tablet.cc:1620] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:05.544904 18828 tablet.cc:1722] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:05.545408 18828 tablet.cc:1620] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:15:05.563644 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=1dc9be11772e41678fea5d654ee4210f] alter complete (version 2)
I20240502 14:15:05.564190 13318 raft_consensus.cc:1261] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:05.564671 18788 consensus_queue.cc:1035] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:05.570487 16022 raft_consensus.cc:1261] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:05.572510 18788 consensus_queue.cc:1035] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:05.576802 13444 raft_consensus.cc:1261] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:05.577422 18788 consensus_queue.cc:1035] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:05.590487 18788 raft_consensus.cc:2932] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:05.596414 13318 raft_consensus.cc:2932] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:05.603762 13044 catalog_manager.cc:5484] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:05.604534 18821 tablet.cc:1722] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:05.605020 18821 tablet.cc:1620] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:05.608875 16284 consensus_queue.cc:237] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:05.611825 13447 raft_consensus.cc:1261] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:05.612538 18785 consensus_queue.cc:1035] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:05.613545 13447 raft_consensus.cc:2932] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:05.632848 13319 raft_consensus.cc:1261] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:05.633394 18780 raft_consensus.cc:2932] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:05.634949 18604 consensus_queue.cc:1035] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:05.636869 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet caa4af5f41674aaa8587158f25e495fb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:05.636426 13319 raft_consensus.cc:2932] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:05.640129 13444 raft_consensus.cc:2932] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:05.645388 13042 catalog_manager.cc:5484] T caa4af5f41674aaa8587158f25e495fb P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:05.651774 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet caa4af5f41674aaa8587158f25e495fb with delete_type TABLET_DATA_TOMBSTONED (TS d1da7f11f0604f78934354a992bb0f55 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:05.655288 13044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:05.660579 18855 tablet_replica.cc:330] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:05.660872 18855 raft_consensus.cc:2229] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:05.661177 18855 pending_rounds.cc:62] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55: Trying to abort 1 pending ops.
I20240502 14:15:05.661373 18855 pending_rounds.cc:68] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023244515454533632 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "caa4af5f41674aaa8587158f25e495fb" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } } }
I20240502 14:15:05.662215 18855 raft_consensus.cc:2894] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 14:15:05.662367 18855 raft_consensus.cc:2866] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:15:05.662489 18855 raft_consensus.cc:2258] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:05.663352 18855 ts_tablet_manager.cc:1875] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:05.665299 16134 tablet_service.cc:1467] Processing CreateTablet for tablet 1c251cfdbec94259be5afbb1b7dbaa38 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=fc31add041984706b1347b9ccdaecbaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.665709 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c251cfdbec94259be5afbb1b7dbaa38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.667446 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 1c251cfdbec94259be5afbb1b7dbaa38 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=fc31add041984706b1347b9ccdaecbaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.667793 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c251cfdbec94259be5afbb1b7dbaa38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.669865 13427 tablet_service.cc:1467] Processing CreateTablet for tablet cee69dfc4ded45de81cfd8b06c5f2f2f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=fc31add041984706b1347b9ccdaecbaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.670068 13429 tablet_service.cc:1467] Processing CreateTablet for tablet df55a71945d34af0affa6eccbdf84dae (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=fc31add041984706b1347b9ccdaecbaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.670086 13560 tablet_service.cc:1467] Processing CreateTablet for tablet cee69dfc4ded45de81cfd8b06c5f2f2f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=fc31add041984706b1347b9ccdaecbaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.670274 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee69dfc4ded45de81cfd8b06c5f2f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.670424 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee69dfc4ded45de81cfd8b06c5f2f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.670444 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df55a71945d34af0affa6eccbdf84dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.673974 16135 tablet_service.cc:1467] Processing CreateTablet for tablet df55a71945d34af0affa6eccbdf84dae (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=fc31add041984706b1347b9ccdaecbaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.674336 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df55a71945d34af0affa6eccbdf84dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.677567 15999 tablet_service.cc:1467] Processing CreateTablet for tablet 1c251cfdbec94259be5afbb1b7dbaa38 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=fc31add041984706b1347b9ccdaecbaf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.677958 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c251cfdbec94259be5afbb1b7dbaa38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.678046 16000 tablet_service.cc:1467] Processing CreateTablet for tablet cee69dfc4ded45de81cfd8b06c5f2f2f (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=fc31add041984706b1347b9ccdaecbaf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.678350 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee69dfc4ded45de81cfd8b06c5f2f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.680125 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:05.686986 16002 tablet_service.cc:1467] Processing CreateTablet for tablet df55a71945d34af0affa6eccbdf84dae (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=fc31add041984706b1347b9ccdaecbaf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:05.687341 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df55a71945d34af0affa6eccbdf84dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.692529 18820 tablet.cc:1722] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:05.693125 18821 tablet.cc:1722] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:05.698236 18857 tablet_bootstrap.cc:492] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:05.700230 18857 tablet_bootstrap.cc:654] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.702006 18859 tablet_bootstrap.cc:492] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:05.704242 18859 tablet_bootstrap.cc:654] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.713323 18857 tablet_bootstrap.cc:492] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:05.713515 18857 ts_tablet_manager.cc:1367] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.003s
I20240502 14:15:05.714322 18857 raft_consensus.cc:348] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:05.714612 18857 raft_consensus.cc:374] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.714815 18857 raft_consensus.cc:729] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.715101 18857 consensus_queue.cc:260] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:05.715848 18857 ts_tablet_manager.cc:1398] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:05.716161 18857 tablet_bootstrap.cc:492] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:05.717720 18857 tablet_bootstrap.cc:654] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.727644 18828 tablet.cc:1722] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:05.734589 18858 tablet_bootstrap.cc:492] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:05.736642 18858 tablet_bootstrap.cc:654] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.738392 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=1dc9be11772e41678fea5d654ee4210f] alter complete (version 3)
I20240502 14:15:05.752899 18857 tablet_bootstrap.cc:492] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:05.753103 18857 ts_tablet_manager.cc:1367] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.008s
I20240502 14:15:05.753903 18857 raft_consensus.cc:348] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:05.754168 18857 raft_consensus.cc:374] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.754256 18857 raft_consensus.cc:729] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.754515 18857 consensus_queue.cc:260] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:05.755146 18857 ts_tablet_manager.cc:1398] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:05.757077 16155 raft_consensus.cc:1203] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:05.758265 18866 tablet_bootstrap.cc:492] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:05.767089 18866 tablet_bootstrap.cc:654] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.769412 18858 tablet_bootstrap.cc:492] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:05.770838 18858 ts_tablet_manager.cc:1367] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.004s
I20240502 14:15:05.771739 18858 raft_consensus.cc:348] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:05.772747 18858 raft_consensus.cc:374] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.772974 18858 raft_consensus.cc:729] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.773317 18858 consensus_queue.cc:260] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:05.775256 18866 tablet_bootstrap.cc:492] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:05.775614 18858 ts_tablet_manager.cc:1398] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:15:05.775885 18866 ts_tablet_manager.cc:1367] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20240502 14:15:05.776273 18858 tablet_bootstrap.cc:492] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:05.783385 18858 tablet_bootstrap.cc:654] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.785451 18866 raft_consensus.cc:348] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:05.786811 18866 raft_consensus.cc:374] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.809351 18856 tablet_bootstrap.cc:492] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:05.787067 18866 raft_consensus.cc:729] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.812392 18866 consensus_queue.cc:260] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:05.813030 18866 ts_tablet_manager.cc:1398] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.037s user 0.001s sys 0.000s
I20240502 14:15:05.816892 18855 ts_tablet_manager.cc:1888] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:05.817286 18856 tablet_bootstrap.cc:654] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.818289 18858 tablet_bootstrap.cc:492] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:05.818531 18858 ts_tablet_manager.cc:1367] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.017s
I20240502 14:15:05.820082 18855 log.cc:1196] T caa4af5f41674aaa8587158f25e495fb P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/caa4af5f41674aaa8587158f25e495fb
I20240502 14:15:05.821249 18858 raft_consensus.cc:348] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:05.821530 18858 raft_consensus.cc:374] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.821619 18858 raft_consensus.cc:729] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.821861 18858 consensus_queue.cc:260] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:05.822577 18858 ts_tablet_manager.cc:1398] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:15:05.837389 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet caa4af5f41674aaa8587158f25e495fb (table rebalancer_extra_table_4 [id=cb136ab1e407401daef8e9856e8ee6f9]) successfully deleted
I20240502 14:15:05.837538 18856 tablet_bootstrap.cc:492] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:05.837705 18856 ts_tablet_manager.cc:1367] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s
I20240502 14:15:05.842181 18856 raft_consensus.cc:348] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:05.842502 18856 raft_consensus.cc:374] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.842613 18856 raft_consensus.cc:729] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.842888 18856 consensus_queue.cc:260] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:05.847931 13044 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:05.848176 13044 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:05.848421 18856 ts_tablet_manager.cc:1398] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:15:05.849273 18859 tablet_bootstrap.cc:492] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:05.849505 18859 ts_tablet_manager.cc:1367] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.148s user 0.013s sys 0.063s
I20240502 14:15:05.850965 18859 raft_consensus.cc:348] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:05.852097 18859 raft_consensus.cc:374] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:05.852252 18859 raft_consensus.cc:729] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:05.852558 18859 consensus_queue.cc:260] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:05.855049 18859 ts_tablet_manager.cc:1398] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:15:05.855427 18859 tablet_bootstrap.cc:492] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:05.857230 13044 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet fd2f7636fde94bf0b1790dbcc14332d4
I20240502 14:15:05.857286 18849 raft_consensus.cc:482] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:05.857401 18849 raft_consensus.cc:504] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:05.857635 18859 tablet_bootstrap.cc:654] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:05.857923 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet fd2f7636fde94bf0b1790dbcc14332d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:05 UTC) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:05.858044 18849 leader_election.cc:288] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:05.861594 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee69dfc4ded45de81cfd8b06c5f2f2f" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:05.861809 16022 raft_consensus.cc:2454] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:05.862310 13516 leader_election.cc:302] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:05.863076 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet fd2f7636fde94bf0b1790dbcc14332d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:05 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:05.863296 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee69dfc4ded45de81cfd8b06c5f2f2f" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:05.863555 13444 raft_consensus.cc:2454] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:05.863903 18731 tablet_replica.cc:330] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:05.864145 18731 raft_consensus.cc:2229] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:05.864555 18731 raft_consensus.cc:2258] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:05.864861 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet fd2f7636fde94bf0b1790dbcc14332d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:05 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:15:05.867961 18849 raft_consensus.cc:2781] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:05.868052 18849 raft_consensus.cc:482] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:05.868139 18849 raft_consensus.cc:3037] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.873497 18731 ts_tablet_manager.cc:1875] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:05.887517 18788 raft_consensus.cc:482] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:05.887728 18788 raft_consensus.cc:504] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:05.888495 18788 leader_election.cc:288] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:05.889456 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c251cfdbec94259be5afbb1b7dbaa38" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:05.889657 16152 raft_consensus.cc:2454] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:05.890126 16220 leader_election.cc:302] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:05.890581 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c251cfdbec94259be5afbb1b7dbaa38" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
W20240502 14:15:05.891098 16221 leader_election.cc:341] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:05.891458 18788 raft_consensus.cc:2781] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:05.891530 18788 raft_consensus.cc:482] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:05.891613 18788 raft_consensus.cc:3037] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.871119 18849 raft_consensus.cc:504] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:05.894511 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee69dfc4ded45de81cfd8b06c5f2f2f" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:05.894501 18788 raft_consensus.cc:504] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:05.894726 13444 raft_consensus.cc:3037] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.895453 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c251cfdbec94259be5afbb1b7dbaa38" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
W20240502 14:15:05.895905 16221 leader_election.cc:341] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:05.896566 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c251cfdbec94259be5afbb1b7dbaa38" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:05.896733 16152 raft_consensus.cc:3037] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.897061 13444 raft_consensus.cc:2454] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:05.897995 13514 leader_election.cc:302] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:05.898309 18518 raft_consensus.cc:2781] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:05.899500 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee69dfc4ded45de81cfd8b06c5f2f2f" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:05.899721 16022 raft_consensus.cc:3037] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:05.902446 16022 raft_consensus.cc:2454] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:05.902480 16152 raft_consensus.cc:2454] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:05.902920 16220 leader_election.cc:302] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters: d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:05.903295 18783 raft_consensus.cc:2781] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:05.903466 18783 raft_consensus.cc:686] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:05.903791 18783 consensus_queue.cc:237] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:05.904160 18788 leader_election.cc:288] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:05.905511 18518 raft_consensus.cc:686] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:15:05.905776 18518 consensus_queue.cc:237] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:05.915794 18849 leader_election.cc:288] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:05.923012 18731 ts_tablet_manager.cc:1888] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:05.923200 18731 log.cc:1196] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/fd2f7636fde94bf0b1790dbcc14332d4
I20240502 14:15:05.923755 18731 ts_tablet_manager.cc:1909] T fd2f7636fde94bf0b1790dbcc14332d4 P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:15:05.940090 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet fd2f7636fde94bf0b1790dbcc14332d4 (table rebalancer_dynamic_table_new_name [id=1dc9be11772e41678fea5d654ee4210f]) successfully deleted
I20240502 14:15:05.955680 18877 tablet_replica.cc:330] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:15:05.955945 18877 raft_consensus.cc:2229] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:05.956259 18877 raft_consensus.cc:2258] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:05.962085 13042 catalog_manager.cc:5484] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:05.968734 18875 tablet_replica.cc:330] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:05.969369 13044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:05.970445 18877 ts_tablet_manager.cc:1875] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:05.970777 18875 raft_consensus.cc:2229] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:05.971124 18875 raft_consensus.cc:2258] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:05.984954 16266 tablet_service.cc:1467] Processing CreateTablet for tablet bd52763c9e5a4492a81e9cf62011b609 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=73c8d57abf7240968438c71f280bbc82]), partition=
I20240502 14:15:05.985462 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd52763c9e5a4492a81e9cf62011b609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.990593 16134 tablet_service.cc:1467] Processing CreateTablet for tablet bd52763c9e5a4492a81e9cf62011b609 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=73c8d57abf7240968438c71f280bbc82]), partition=
I20240502 14:15:05.990968 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd52763c9e5a4492a81e9cf62011b609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:05.996438 18856 tablet_bootstrap.cc:492] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:05.997941 18856 tablet_bootstrap.cc:654] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.011757 18858 tablet_bootstrap.cc:492] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:06.013239 18858 tablet_bootstrap.cc:654] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.030490 18858 tablet_bootstrap.cc:492] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:06.030778 18858 ts_tablet_manager.cc:1367] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.007s
I20240502 14:15:06.031807 18858 raft_consensus.cc:348] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:06.032217 18858 raft_consensus.cc:374] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.032395 18858 raft_consensus.cc:729] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.032701 18858 consensus_queue.cc:260] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:05.990209 16401 tablet_service.cc:1467] Processing CreateTablet for tablet bd52763c9e5a4492a81e9cf62011b609 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=73c8d57abf7240968438c71f280bbc82]), partition=
I20240502 14:15:06.038039 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd52763c9e5a4492a81e9cf62011b609. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.038722 18875 ts_tablet_manager.cc:1875] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:06.044692 18858 ts_tablet_manager.cc:1398] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.014s user 0.001s sys 0.006s
I20240502 14:15:06.049314 13044 catalog_manager.cc:5484] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:06.055346 18856 tablet_bootstrap.cc:492] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:06.055559 18856 ts_tablet_manager.cc:1367] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.059s user 0.000s sys 0.012s
I20240502 14:15:06.058193 18856 raft_consensus.cc:348] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:06.059453 18856 raft_consensus.cc:374] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.060149 18856 raft_consensus.cc:729] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
W20240502 14:15:06.060509 16089 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.200:43857, user_credentials={real_user=slave}} blocked reactor thread for 61489.4us
I20240502 14:15:06.062716 18856 consensus_queue.cc:260] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:06.065281 18856 ts_tablet_manager.cc:1398] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:15:06.077337 18889 tablet_bootstrap.cc:492] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:06.083024 18859 tablet_bootstrap.cc:492] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:06.083307 18859 ts_tablet_manager.cc:1367] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.228s user 0.073s sys 0.102s
I20240502 14:15:06.084354 18859 raft_consensus.cc:348] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:06.084586 18859 raft_consensus.cc:374] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.084651 18859 raft_consensus.cc:729] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.084918 18859 consensus_queue.cc:260] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:06.086822 18859 ts_tablet_manager.cc:1398] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240502 14:15:06.087117 18859 tablet_bootstrap.cc:492] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:06.090412 18859 tablet_bootstrap.cc:654] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.086215 18877 ts_tablet_manager.cc:1888] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:06.091315 18877 log.cc:1196] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/fd2f7636fde94bf0b1790dbcc14332d4
I20240502 14:15:06.091795 18877 ts_tablet_manager.cc:1909] T fd2f7636fde94bf0b1790dbcc14332d4 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:15:06.095990 18889 tablet_bootstrap.cc:654] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.098990 18859 tablet_bootstrap.cc:492] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:06.099190 18859 ts_tablet_manager.cc:1367] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20240502 14:15:06.101521 18859 raft_consensus.cc:348] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:06.101780 18859 raft_consensus.cc:374] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.101912 18859 raft_consensus.cc:729] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.102173 18859 consensus_queue.cc:260] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:06.103914 18859 ts_tablet_manager.cc:1398] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:15:06.109637 18783 raft_consensus.cc:482] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.109841 18783 raft_consensus.cc:504] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:06.110617 18783 leader_election.cc:288] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:06.111163 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd52763c9e5a4492a81e9cf62011b609" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
W20240502 14:15:06.111944 16220 leader_election.cc:341] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:06.113330 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd52763c9e5a4492a81e9cf62011b609" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:06.113653 16152 raft_consensus.cc:2454] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:06.115738 16220 leader_election.cc:302] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:06.116397 18783 raft_consensus.cc:2781] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:06.116561 18783 raft_consensus.cc:482] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:06.116731 18783 raft_consensus.cc:3037] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.117468 18875 ts_tablet_manager.cc:1888] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:06.117666 18875 log.cc:1196] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/fd2f7636fde94bf0b1790dbcc14332d4
I20240502 14:15:06.118273 18875 ts_tablet_manager.cc:1909] T fd2f7636fde94bf0b1790dbcc14332d4 P 8b47d4390f8343a493d6c00123785968: Deleting consensus metadata
I20240502 14:15:06.119311 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet fd2f7636fde94bf0b1790dbcc14332d4 (table rebalancer_dynamic_table_new_name [id=1dc9be11772e41678fea5d654ee4210f]) successfully deleted
I20240502 14:15:06.120390 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet fd2f7636fde94bf0b1790dbcc14332d4 (table rebalancer_dynamic_table_new_name [id=1dc9be11772e41678fea5d654ee4210f]) successfully deleted
I20240502 14:15:06.121760 18783 raft_consensus.cc:504] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:06.122929 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd52763c9e5a4492a81e9cf62011b609" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
W20240502 14:15:06.123553 16220 leader_election.cc:341] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:06.124002 18783 leader_election.cc:288] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:06.124478 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd52763c9e5a4492a81e9cf62011b609" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:06.124677 16152 raft_consensus.cc:3037] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.128440 18885 raft_consensus.cc:482] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.128554 18885 raft_consensus.cc:504] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:06.129015 18889 tablet_bootstrap.cc:492] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:06.129194 18889 ts_tablet_manager.cc:1367] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20240502 14:15:06.129339 18885 leader_election.cc:288] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:06.130128 18889 raft_consensus.cc:348] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:06.130342 18889 raft_consensus.cc:374] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.130440 18889 raft_consensus.cc:729] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.130668 18889 consensus_queue.cc:260] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:06.131244 18889 ts_tablet_manager.cc:1398] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:06.131898 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df55a71945d34af0affa6eccbdf84dae" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:06.132193 16022 raft_consensus.cc:2454] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:06.132473 16152 raft_consensus.cc:2454] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:06.132560 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df55a71945d34af0affa6eccbdf84dae" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:06.132750 13444 raft_consensus.cc:2454] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:06.132799 16220 leader_election.cc:302] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:06.133726 18783 raft_consensus.cc:2781] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:06.133929 18783 raft_consensus.cc:686] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:06.134193 18783 consensus_queue.cc:237] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:06.135843 16090 leader_election.cc:302] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:06.136451 18887 raft_consensus.cc:2781] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:06.136556 18887 raft_consensus.cc:482] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:06.136687 18887 raft_consensus.cc:3037] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.139299 18887 raft_consensus.cc:504] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:06.140420 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df55a71945d34af0affa6eccbdf84dae" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:06.140611 13444 raft_consensus.cc:3037] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.143085 13444 raft_consensus.cc:2454] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:06.144371 16088 leader_election.cc:302] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:06.144624 18885 raft_consensus.cc:2781] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:06.144760 18885 raft_consensus.cc:686] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:06.145020 18885 consensus_queue.cc:237] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:06.146546 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df55a71945d34af0affa6eccbdf84dae" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:06.146728 16022 raft_consensus.cc:3037] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.150251 18887 leader_election.cc:288] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:06.149205 16022 raft_consensus.cc:2454] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:06.161079 13044 catalog_manager.cc:5484] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:06.162578 13042 catalog_manager.cc:5484] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:06.269210 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:06.277956 16420 raft_consensus.cc:3037] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.280566 16420 raft_consensus.cc:1261] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:06.281209 16152 raft_consensus.cc:1261] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:06.281183 18788 consensus_queue.cc:1035] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.282213 18788 consensus_queue.cc:1035] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.298820 16020 raft_consensus.cc:3037] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.307381 18604 consensus_queue.cc:1035] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.308332 13042 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet bd52763c9e5a4492a81e9cf62011b609 (table rebalancer_dynamic_table [id=73c8d57abf7240968438c71f280bbc82]). Expected version 1 got 0
I20240502 14:15:06.321746 18930 tablet.cc:1722] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:06.322654 18930 tablet.cc:1620] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:06.324965 13042 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet bd52763c9e5a4492a81e9cf62011b609 (table rebalancer_dynamic_table [id=73c8d57abf7240968438c71f280bbc82]). Expected version 1 got 0
I20240502 14:15:06.336716 13044 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet bd52763c9e5a4492a81e9cf62011b609 (table rebalancer_dynamic_table [id=73c8d57abf7240968438c71f280bbc82]). Expected version 1 got 0
I20240502 14:15:06.340785 18932 tablet.cc:1722] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:06.341495 18932 tablet.cc:1620] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:06.345247 13045 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet bd52763c9e5a4492a81e9cf62011b609 (table rebalancer_dynamic_table [id=73c8d57abf7240968438c71f280bbc82]). Expected version 1 got 0
I20240502 14:15:06.346097 18788 consensus_queue.cc:1035] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.347398 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=73c8d57abf7240968438c71f280bbc82] alter complete (version 1)
I20240502 14:15:06.357352 18814 raft_consensus.cc:1050] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER c86f7440570f4796babfd44c3d7fd847 to VOTER
I20240502 14:15:06.358146 18814 consensus_queue.cc:237] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:06.364550 13318 raft_consensus.cc:1261] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:06.365161 18167 consensus_queue.cc:1035] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:06.367118 16152 raft_consensus.cc:1261] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:06.367579 18798 consensus_queue.cc:1035] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:06.367672 13187 raft_consensus.cc:1261] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:06.369071 18798 consensus_queue.cc:1035] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:06.372116 18928 tablet.cc:1722] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:06.372792 18928 tablet.cc:1620] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:06.375241 18798 raft_consensus.cc:2932] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:06.380082 16151 raft_consensus.cc:2932] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:06.396857 13318 raft_consensus.cc:2932] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:06.409446 13188 raft_consensus.cc:2932] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:06.412674 13044 catalog_manager.cc:5484] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:06.451181 18881 consensus_queue.cc:1035] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.463335 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:06.473205 18928 tablet.cc:1722] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:06.473673 18928 tablet.cc:1620] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:06.473927 18930 tablet.cc:1722] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:06.474319 18930 tablet.cc:1620] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:06.477532 18849 consensus_queue.cc:1035] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.480088 18932 tablet.cc:1722] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:06.480515 18932 tablet.cc:1620] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:06.487612 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=73c8d57abf7240968438c71f280bbc82] alter complete (version 2)
I20240502 14:15:06.511229 13447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0e393a7736b14d5483973f9cb89058f8"
dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51110
I20240502 14:15:06.511533 13447 raft_consensus.cc:595] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:06.538061 13447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7852539de07744caaf385620c5166861"
dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51132
I20240502 14:15:06.538218 13447 raft_consensus.cc:595] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:06.551424 13187 consensus_queue.cc:237] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:06.553766 16020 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:15:06.553869 13121 consensus_peers.cc:485] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2905b7940fb45c49ef8b18943dae256. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:06.554008 13318 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:06.554556 18992 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:06.554876 18449 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:06.557332 18993 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:06.558013 13318 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:06.558048 16020 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:06.562711 13042 catalog_manager.cc:5484] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:06.578498 13447 consensus_queue.cc:237] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:06.580849 13318 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:15:06.581015 13384 consensus_peers.cc:485] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c5329f38f3343baaf43265990d86938. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:06.581635 13187 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:06.581938 18798 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:06.582334 18167 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:06.584436 18798 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:06.584959 13318 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:06.587781 13187 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:06.589190 13036 catalog_manager.cc:5484] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:06.591681 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:06.603353 18928 tablet.cc:1722] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:06.604555 18930 tablet.cc:1722] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:06.606027 18932 tablet.cc:1722] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:06.612294 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=73c8d57abf7240968438c71f280bbc82] alter complete (version 3)
I20240502 14:15:06.615442 16284 consensus_queue.cc:237] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:06.617596 16418 raft_consensus.cc:1261] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:06.618100 18783 consensus_queue.cc:1035] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:06.618216 16020 raft_consensus.cc:1261] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:06.618628 18783 consensus_queue.cc:1035] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:06.618955 16219 consensus_peers.cc:485] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7da975869e82479b91fdda30680e6b3c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:06.621081 18604 raft_consensus.cc:2932] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:06.621747 16418 raft_consensus.cc:2932] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:06.628191 13042 catalog_manager.cc:5484] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:06.635084 16020 raft_consensus.cc:2932] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:06.654852 13318 consensus_queue.cc:237] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:06.656829 16020 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:15:06.657037 13252 consensus_peers.cc:485] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d5975e02205a43d5b93ed8ab51f8a0f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:06.657681 13579 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:06.658149 18737 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:06.658424 18413 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:06.658854 18798 raft_consensus.cc:979] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3: : Instructing follower c86f7440570f4796babfd44c3d7fd847 to start an election
I20240502 14:15:06.659006 18798 raft_consensus.cc:1067] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Signalling peer c86f7440570f4796babfd44c3d7fd847 to start an election
I20240502 14:15:06.660056 16155 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0e393a7736b14d5483973f9cb89058f8"
dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
from {username='slave'} at 127.6.196.195:52943
I20240502 14:15:06.660328 16155 raft_consensus.cc:482] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:06.660454 16155 raft_consensus.cc:3037] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:06.661638 18413 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:06.663065 16155 raft_consensus.cc:504] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:06.664283 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e393a7736b14d5483973f9cb89058f8" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:06.664448 13318 raft_consensus.cc:3037] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:06.665557 16022 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:06.668776 16155 leader_election.cc:288] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 2 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:06.669045 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e393a7736b14d5483973f9cb89058f8" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:06.669277 13444 raft_consensus.cc:3032] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:06.669353 13444 raft_consensus.cc:729] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:06.669620 13444 consensus_queue.cc:260] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:06.670110 13444 raft_consensus.cc:3037] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:06.670316 13318 raft_consensus.cc:2454] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 2.
I20240502 14:15:06.670701 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e393a7736b14d5483973f9cb89058f8" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:06.670892 13580 raft_consensus.cc:3037] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:06.671929 13444 raft_consensus.cc:2454] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 2.
I20240502 14:15:06.672600 13580 raft_consensus.cc:2454] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 2.
I20240502 14:15:06.673050 16088 leader_election.cc:302] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:06.673346 18885 raft_consensus.cc:2781] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:06.681578 13044 catalog_manager.cc:5484] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:06.694955 13579 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:06.705318 18885 raft_consensus.cc:686] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 2 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:06.705659 18885 consensus_queue.cc:237] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:06.707800 13318 consensus_queue.cc:237] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:06.709389 13044 catalog_manager.cc:5484] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 1 to 2, leader changed from 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 2 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:06.710505 13187 raft_consensus.cc:1261] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:15:06.710572 13252 consensus_peers.cc:485] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6395ce4c30eb4644b0a711de3e60b9c0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:06.710875 13444 raft_consensus.cc:1261] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:06.711345 19023 consensus_queue.cc:1035] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:06.711807 19023 consensus_queue.cc:1035] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:06.716212 13044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:06.717595 13042 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:06.717790 13042 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:06.725503 13042 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet bd52763c9e5a4492a81e9cf62011b609
I20240502 14:15:06.726503 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet bd52763c9e5a4492a81e9cf62011b609 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:06.726532 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet bd52763c9e5a4492a81e9cf62011b609 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:06 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:06.727655 19023 raft_consensus.cc:2932] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:06.728217 16134 tablet_service.cc:1514] Processing DeleteTablet for tablet bd52763c9e5a4492a81e9cf62011b609 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:06 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:06.730906 13444 raft_consensus.cc:2932] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:06.732671 13188 raft_consensus.cc:2932] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:06.737030 16400 tablet_service.cc:1467] Processing CreateTablet for tablet 748ccda9cdc94ebaaa432562e20f9fa3 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=fc0747d6473d42628aa0af6e2ee96269]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.737280 13427 tablet_service.cc:1467] Processing CreateTablet for tablet c5652dfa60d84803abbac9201e5874db (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=fc0747d6473d42628aa0af6e2ee96269]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.737542 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748ccda9cdc94ebaaa432562e20f9fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.737622 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5652dfa60d84803abbac9201e5874db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.741569 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 748ccda9cdc94ebaaa432562e20f9fa3 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=fc0747d6473d42628aa0af6e2ee96269]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.741817 15999 tablet_service.cc:1467] Processing CreateTablet for tablet 748ccda9cdc94ebaaa432562e20f9fa3 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=fc0747d6473d42628aa0af6e2ee96269]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.741962 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748ccda9cdc94ebaaa432562e20f9fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.741910 16000 tablet_service.cc:1467] Processing CreateTablet for tablet c5652dfa60d84803abbac9201e5874db (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=fc0747d6473d42628aa0af6e2ee96269]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.742226 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 748ccda9cdc94ebaaa432562e20f9fa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.742404 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5652dfa60d84803abbac9201e5874db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.743031 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 6931796f283a4f20b7fec8c3163b8444 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=fc0747d6473d42628aa0af6e2ee96269]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.743382 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6931796f283a4f20b7fec8c3163b8444. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.745594 16265 tablet_service.cc:1467] Processing CreateTablet for tablet c5652dfa60d84803abbac9201e5874db (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=fc0747d6473d42628aa0af6e2ee96269]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.745954 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5652dfa60d84803abbac9201e5874db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.750154 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 6931796f283a4f20b7fec8c3163b8444 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=fc0747d6473d42628aa0af6e2ee96269]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.750579 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6931796f283a4f20b7fec8c3163b8444. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.755196 19021 consensus_queue.cc:1035] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.755477 19020 consensus_queue.cc:1035] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:06.756295 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 6931796f283a4f20b7fec8c3163b8444 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=fc0747d6473d42628aa0af6e2ee96269]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:06.756649 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6931796f283a4f20b7fec8c3163b8444. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.769775 13045 catalog_manager.cc:5484] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 4, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:06.785204 19036 tablet_replica.cc:330] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:06.785499 19036 raft_consensus.cc:2229] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:06.785773 19036 raft_consensus.cc:2258] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:06.786824 19036 ts_tablet_manager.cc:1875] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:06.789115 19037 tablet_bootstrap.cc:492] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:06.790234 19041 tablet_bootstrap.cc:492] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:06.791580 19043 tablet_bootstrap.cc:492] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:06.791780 19040 tablet_bootstrap.cc:492] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:06.791570 19044 tablet_bootstrap.cc:492] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:06.793066 19043 tablet_bootstrap.cc:654] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.793941 19040 tablet_bootstrap.cc:654] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.794355 19038 tablet_bootstrap.cc:492] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:06.794780 19041 tablet_bootstrap.cc:654] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.795269 19034 tablet_replica.cc:330] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:06.796474 19035 tablet_replica.cc:330] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:06.796706 19035 raft_consensus.cc:2229] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:06.796906 19035 raft_consensus.cc:2258] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:06.796960 19038 tablet_bootstrap.cc:654] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.797458 19044 tablet_bootstrap.cc:654] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.797518 19034 raft_consensus.cc:2229] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:06.798182 19034 raft_consensus.cc:2258] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:06.798745 19043 tablet_bootstrap.cc:492] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:06.798763 19037 tablet_bootstrap.cc:654] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.799343 19043 ts_tablet_manager.cc:1367] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240502 14:15:06.800201 19034 ts_tablet_manager.cc:1875] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:06.800348 19035 ts_tablet_manager.cc:1875] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:06.803207 19042 tablet_bootstrap.cc:492] T 748ccda9cdc94ebaaa432562e20f9fa3 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:06.804399 19044 tablet_bootstrap.cc:492] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:06.804903 19042 tablet_bootstrap.cc:654] T 748ccda9cdc94ebaaa432562e20f9fa3 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.805600 19044 ts_tablet_manager.cc:1367] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 14:15:06.801978 19043 raft_consensus.cc:348] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:06.806983 19041 tablet_bootstrap.cc:492] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:06.807149 19041 ts_tablet_manager.cc:1367] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240502 14:15:06.807835 19041 raft_consensus.cc:348] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:06.807966 19040 tablet_bootstrap.cc:492] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:06.808105 19040 ts_tablet_manager.cc:1367] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20240502 14:15:06.808310 19043 raft_consensus.cc:374] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.808135 19044 raft_consensus.cc:348] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.808404 19044 raft_consensus.cc:374] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.808487 19044 raft_consensus.cc:729] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.808441 19043 raft_consensus.cc:729] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.808879 19041 raft_consensus.cc:374] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.808997 19041 raft_consensus.cc:729] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.809270 19041 consensus_queue.cc:260] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:06.808719 19044 consensus_queue.cc:260] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.811225 19040 raft_consensus.cc:348] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:06.811674 19044 ts_tablet_manager.cc:1398] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:15:06.811929 19041 ts_tablet_manager.cc:1398] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:15:06.813992 19043 consensus_queue.cc:260] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:06.815284 19040 raft_consensus.cc:374] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.815445 19040 raft_consensus.cc:729] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.816540 19040 consensus_queue.cc:260] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:06.817242 19040 ts_tablet_manager.cc:1398] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 14:15:06.820070 19036 ts_tablet_manager.cc:1888] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:06.822178 19040 tablet_bootstrap.cc:492] T 748ccda9cdc94ebaaa432562e20f9fa3 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:06.825944 19040 tablet_bootstrap.cc:654] T 748ccda9cdc94ebaaa432562e20f9fa3 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.829690 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:06.827596 19035 ts_tablet_manager.cc:1888] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:06.830531 19035 log.cc:1196] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/bd52763c9e5a4492a81e9cf62011b609
I20240502 14:15:06.831737 19036 log.cc:1196] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/bd52763c9e5a4492a81e9cf62011b609
I20240502 14:15:06.832264 19036 ts_tablet_manager.cc:1909] T bd52763c9e5a4492a81e9cf62011b609 P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:15:06.832815 19043 ts_tablet_manager.cc:1398] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.032s user 0.001s sys 0.000s
I20240502 14:15:06.833331 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet bd52763c9e5a4492a81e9cf62011b609 (table rebalancer_dynamic_table_new_name [id=73c8d57abf7240968438c71f280bbc82]) successfully deleted
I20240502 14:15:06.834645 19035 ts_tablet_manager.cc:1909] T bd52763c9e5a4492a81e9cf62011b609 P 8b47d4390f8343a493d6c00123785968: Deleting consensus metadata
I20240502 14:15:06.836190 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet bd52763c9e5a4492a81e9cf62011b609 (table rebalancer_dynamic_table_new_name [id=73c8d57abf7240968438c71f280bbc82]) successfully deleted
I20240502 14:15:06.840736 19042 tablet_bootstrap.cc:492] T 748ccda9cdc94ebaaa432562e20f9fa3 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:06.840942 19042 ts_tablet_manager.cc:1367] T 748ccda9cdc94ebaaa432562e20f9fa3 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.000s
I20240502 14:15:06.842125 19038 tablet_bootstrap.cc:492] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:06.842305 19038 ts_tablet_manager.cc:1367] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.048s user 0.008s sys 0.000s
I20240502 14:15:06.842208 19042 raft_consensus.cc:348] T 748ccda9cdc94ebaaa432562e20f9fa3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.842451 19042 raft_consensus.cc:374] T 748ccda9cdc94ebaaa432562e20f9fa3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.842569 19042 raft_consensus.cc:729] T 748ccda9cdc94ebaaa432562e20f9fa3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.842823 19042 consensus_queue.cc:260] T 748ccda9cdc94ebaaa432562e20f9fa3 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.843005 18814 raft_consensus.cc:979] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: : Instructing follower c86f7440570f4796babfd44c3d7fd847 to start an election
I20240502 14:15:06.843132 18814 raft_consensus.cc:1067] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Signalling peer c86f7440570f4796babfd44c3d7fd847 to start an election
I20240502 14:15:06.843839 19038 raft_consensus.cc:348] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.844074 19038 raft_consensus.cc:374] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.844162 19038 raft_consensus.cc:729] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.844259 19042 ts_tablet_manager.cc:1398] T 748ccda9cdc94ebaaa432562e20f9fa3 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:15:06.845288 16155 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7852539de07744caaf385620c5166861"
dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
from {username='slave'} at 127.6.196.195:52943
I20240502 14:15:06.845479 16155 raft_consensus.cc:482] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:06.845559 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 97a61b85f4b849b594e696a6e8c1cedf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9074beaa768c472299f9c09e6f7389bc]), partition=
I20240502 14:15:06.845640 16155 raft_consensus.cc:3037] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:06.845921 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a61b85f4b849b594e696a6e8c1cedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.846071 15999 tablet_service.cc:1467] Processing CreateTablet for tablet 97a61b85f4b849b594e696a6e8c1cedf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9074beaa768c472299f9c09e6f7389bc]), partition=
I20240502 14:15:06.846383 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a61b85f4b849b594e696a6e8c1cedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.848757 16155 raft_consensus.cc:504] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:06.849540 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 97a61b85f4b849b594e696a6e8c1cedf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9074beaa768c472299f9c09e6f7389bc]), partition=
I20240502 14:15:06.849860 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a61b85f4b849b594e696a6e8c1cedf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:06.851449 16155 leader_election.cc:288] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 2 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:06.851794 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852539de07744caaf385620c5166861" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:06.851961 13187 raft_consensus.cc:3037] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:06.853425 19034 ts_tablet_manager.cc:1888] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:06.853590 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852539de07744caaf385620c5166861" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:06.853827 13579 raft_consensus.cc:3037] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:06.854806 19034 log.cc:1196] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/bd52763c9e5a4492a81e9cf62011b609
I20240502 14:15:06.855283 19034 ts_tablet_manager.cc:1909] T bd52763c9e5a4492a81e9cf62011b609 P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:15:06.856170 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet bd52763c9e5a4492a81e9cf62011b609 (table rebalancer_dynamic_table_new_name [id=73c8d57abf7240968438c71f280bbc82]) successfully deleted
I20240502 14:15:06.856987 13579 raft_consensus.cc:2454] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 2.
I20240502 14:15:06.863106 13187 raft_consensus.cc:2454] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 2.
I20240502 14:15:06.863536 16088 leader_election.cc:302] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:06.863888 19021 raft_consensus.cc:2781] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:06.864053 19021 raft_consensus.cc:686] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 2 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:06.864326 19021 consensus_queue.cc:237] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:06.868400 13187 raft_consensus.cc:1226] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Rejecting Update request from peer 88b77f6edc1c4a72b548bf47c05704f3 for earlier term 1. Current term is 2. Ops: []
I20240502 14:15:06.870297 19042 tablet_bootstrap.cc:492] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:06.873399 19041 tablet_bootstrap.cc:492] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:06.873586 19042 tablet_bootstrap.cc:654] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.875360 19041 tablet_bootstrap.cc:654] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.876441 18175 raft_consensus.cc:482] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.876579 18175 raft_consensus.cc:504] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:06.877372 18175 leader_election.cc:288] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:06.878137 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6931796f283a4f20b7fec8c3163b8444" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:06.878465 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6931796f283a4f20b7fec8c3163b8444" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:06.878760 13187 raft_consensus.cc:2454] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:15:06.878929 13579 raft_consensus.cc:2454] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:15:06.879829 16353 leader_election.cc:302] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:06.880273 19055 raft_consensus.cc:2781] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:06.880381 19055 raft_consensus.cc:482] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:06.880501 19055 raft_consensus.cc:3037] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.881500 19040 tablet_bootstrap.cc:492] T 748ccda9cdc94ebaaa432562e20f9fa3 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:06.882558 19040 ts_tablet_manager.cc:1367] T 748ccda9cdc94ebaaa432562e20f9fa3 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.002s
I20240502 14:15:06.883064 19037 tablet_bootstrap.cc:492] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:06.883236 19037 ts_tablet_manager.cc:1367] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.094s user 0.006s sys 0.002s
I20240502 14:15:06.884088 19037 raft_consensus.cc:348] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.884353 19037 raft_consensus.cc:374] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.884485 19037 raft_consensus.cc:729] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.884546 19040 raft_consensus.cc:348] T 748ccda9cdc94ebaaa432562e20f9fa3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.884732 19040 raft_consensus.cc:374] T 748ccda9cdc94ebaaa432562e20f9fa3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.884812 19040 raft_consensus.cc:729] T 748ccda9cdc94ebaaa432562e20f9fa3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.884848 19037 consensus_queue.cc:260] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.885658 19040 consensus_queue.cc:260] T 748ccda9cdc94ebaaa432562e20f9fa3 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.886525 19040 ts_tablet_manager.cc:1398] T 748ccda9cdc94ebaaa432562e20f9fa3 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:15:06.887136 19037 ts_tablet_manager.cc:1398] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:15:06.887539 19037 tablet_bootstrap.cc:492] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:06.895323 19037 tablet_bootstrap.cc:654] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.896164 19055 raft_consensus.cc:504] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:06.863188 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852539de07744caaf385620c5166861" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:06.869079 18798 consensus_queue.cc:1046] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: INVALID_TERM, Last received: 1.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:06.897835 18798 raft_consensus.cc:3032] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:06.897949 18798 raft_consensus.cc:729] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:06.898264 18798 consensus_queue.cc:260] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:06.898799 18798 raft_consensus.cc:3037] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:06.905340 13444 raft_consensus.cc:2454] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 2.
I20240502 14:15:06.906229 19038 consensus_queue.cc:260] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.908003 19038 ts_tablet_manager.cc:1398] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.066s user 0.002s sys 0.000s
I20240502 14:15:06.908349 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6931796f283a4f20b7fec8c3163b8444" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:06.908537 13579 raft_consensus.cc:3037] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.908653 19055 leader_election.cc:288] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:06.911424 13579 raft_consensus.cc:2454] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:15:06.912410 16353 leader_election.cc:302] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:06.912998 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6931796f283a4f20b7fec8c3163b8444" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:06.913213 13188 raft_consensus.cc:3037] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.913148 13043 catalog_manager.cc:5484] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 1 to 2, leader changed from 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 2 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:06.913976 19037 tablet_bootstrap.cc:492] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:06.914141 19037 ts_tablet_manager.cc:1367] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.006s
I20240502 14:15:06.914839 19037 raft_consensus.cc:348] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.915060 19037 raft_consensus.cc:374] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.915148 19037 raft_consensus.cc:729] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.915481 19037 consensus_queue.cc:260] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.915598 13188 raft_consensus.cc:2454] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:15:06.917501 18798 raft_consensus.cc:482] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:06.917620 18798 raft_consensus.cc:504] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.918313 18175 raft_consensus.cc:2781] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:06.918359 18798 leader_election.cc:288] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:06.919154 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5652dfa60d84803abbac9201e5874db" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:06.919301 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5652dfa60d84803abbac9201e5874db" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:06.919395 16288 raft_consensus.cc:2454] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:15:06.919462 16020 raft_consensus.cc:2454] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:15:06.919828 13384 leader_election.cc:302] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:06.920138 18798 raft_consensus.cc:2781] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:06.920205 18798 raft_consensus.cc:482] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:06.920264 18798 raft_consensus.cc:3037] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.920771 19042 tablet_bootstrap.cc:492] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:06.921054 19042 ts_tablet_manager.cc:1367] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.007s
I20240502 14:15:06.921204 19037 ts_tablet_manager.cc:1398] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 14:15:06.922055 19042 raft_consensus.cc:348] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:06.922340 19037 tablet_bootstrap.cc:492] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:06.922390 19042 raft_consensus.cc:374] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.922609 19042 raft_consensus.cc:729] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.924145 18798 raft_consensus.cc:504] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.925179 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5652dfa60d84803abbac9201e5874db" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:06.925424 16288 raft_consensus.cc:3037] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.922856 19042 consensus_queue.cc:260] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:06.929186 19041 tablet_bootstrap.cc:492] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:06.929205 19042 ts_tablet_manager.cc:1398] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:15:06.929373 16288 raft_consensus.cc:2454] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:06.929890 13384 leader_election.cc:302] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:06.930214 18167 raft_consensus.cc:2781] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:06.930336 18167 raft_consensus.cc:686] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:06.930611 18167 consensus_queue.cc:237] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:06.933976 19037 tablet_bootstrap.cc:654] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:06.934409 13042 catalog_manager.cc:5484] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:06.937487 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5652dfa60d84803abbac9201e5874db" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:06.937619 19041 ts_tablet_manager.cc:1367] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.003s
I20240502 14:15:06.937726 16020 raft_consensus.cc:3037] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:06.938299 19041 raft_consensus.cc:348] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:06.938493 19041 raft_consensus.cc:374] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.938547 19041 raft_consensus.cc:729] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.940023 19041 consensus_queue.cc:260] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:06.942483 19041 ts_tablet_manager.cc:1398] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:06.946529 18798 leader_election.cc:288] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:06.946920 16020 raft_consensus.cc:2454] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:06.949074 18175 raft_consensus.cc:686] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:15:06.949398 18175 consensus_queue.cc:237] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:06.958331 19037 tablet_bootstrap.cc:492] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:06.958463 19037 ts_tablet_manager.cc:1367] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.036s user 0.011s sys 0.001s
I20240502 14:15:06.959216 19037 raft_consensus.cc:348] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:06.959393 19037 raft_consensus.cc:374] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:06.959470 19037 raft_consensus.cc:729] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:06.959645 19037 consensus_queue.cc:260] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:06.959794 13046 catalog_manager.cc:5484] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:06.960244 19037 ts_tablet_manager.cc:1398] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:06.966801 13579 consensus_queue.cc:237] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
W20240502 14:15:06.978188 13516 consensus_peers.cc:485] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 652e49046212458280095c016a342f47. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:06.978518 16155 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:06.979331 18878 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:06.979710 13444 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:06.981302 18878 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:06.998062 16151 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:06.996838 18878 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:07.013451 13043 catalog_manager.cc:5484] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:07.015900 19072 ts_tablet_manager.cc:902] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:07.030107 13444 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:07.034826 18668 raft_consensus.cc:482] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:07.035027 18668 raft_consensus.cc:504] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:07.036100 18668 leader_election.cc:288] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:07.036218 19072 tablet_copy_client.cc:308] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:15:07.036617 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748ccda9cdc94ebaaa432562e20f9fa3" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:07.036770 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet 9c5329f38f3343baaf43265990d86938 from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:53727)
I20240502 14:15:07.036878 16155 raft_consensus.cc:2454] T 748ccda9cdc94ebaaa432562e20f9fa3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:07.036898 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet 9c5329f38f3343baaf43265990d86938 from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:53727: session id = d1da7f11f0604f78934354a992bb0f55-9c5329f38f3343baaf43265990d86938
I20240502 14:15:07.037463 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748ccda9cdc94ebaaa432562e20f9fa3" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:07.037706 16418 raft_consensus.cc:2454] T 748ccda9cdc94ebaaa432562e20f9fa3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:07.038379 15956 leader_election.cc:302] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:07.038656 18668 raft_consensus.cc:2781] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:07.038796 18668 raft_consensus.cc:482] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:07.038906 18668 raft_consensus.cc:3037] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.039007 13470 tablet_copy_source_session.cc:205] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:07.040153 19072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5329f38f3343baaf43265990d86938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.042027 18668 raft_consensus.cc:504] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:07.042614 18668 leader_election.cc:288] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:07.043730 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748ccda9cdc94ebaaa432562e20f9fa3" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:07.043942 16155 raft_consensus.cc:3037] T 748ccda9cdc94ebaaa432562e20f9fa3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.044131 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "748ccda9cdc94ebaaa432562e20f9fa3" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:07.044351 16418 raft_consensus.cc:3037] T 748ccda9cdc94ebaaa432562e20f9fa3 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.046008 19072 tablet_copy_client.cc:783] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:07.046226 19072 tablet_copy_client.cc:647] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:07.046483 16155 raft_consensus.cc:2454] T 748ccda9cdc94ebaaa432562e20f9fa3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:07.046823 16418 raft_consensus.cc:2454] T 748ccda9cdc94ebaaa432562e20f9fa3 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:07.047420 15956 leader_election.cc:302] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:07.047923 18668 raft_consensus.cc:2781] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:07.049517 19072 tablet_copy_client.cc:523] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:07.053879 18668 raft_consensus.cc:686] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:07.053968 19072 tablet_bootstrap.cc:492] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:07.054812 18668 consensus_queue.cc:237] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:07.058094 13319 consensus_queue.cc:237] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:07.059958 13042 catalog_manager.cc:5484] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:07.063968 16288 raft_consensus.cc:1261] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:07.064193 16155 raft_consensus.cc:1261] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:07.064469 19023 consensus_queue.cc:1035] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:07.064755 18413 consensus_queue.cc:1035] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:07.065078 13251 consensus_peers.cc:485] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1d607f6e65194399844bc14457fe8f15. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:07.067474 19023 raft_consensus.cc:2932] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:07.068284 16288 raft_consensus.cc:2932] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:07.069907 16155 raft_consensus.cc:2932] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:07.074187 13042 catalog_manager.cc:5484] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:07.097440 19090 ts_tablet_manager.cc:902] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:07.099531 19072 tablet_bootstrap.cc:492] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:07.100051 19072 tablet_bootstrap.cc:492] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:15:07.100358 19072 ts_tablet_manager.cc:1367] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.046s user 0.041s sys 0.001s
I20240502 14:15:07.101236 19072 raft_consensus.cc:348] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:07.101595 19072 raft_consensus.cc:729] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:15:07.101831 19072 consensus_queue.cc:260] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:07.103183 19020 raft_consensus.cc:482] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:07.103338 19020 raft_consensus.cc:504] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:07.103930 19020 leader_election.cc:288] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:07.104806 19072 ts_tablet_manager.cc:1398] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240502 14:15:07.105170 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61b85f4b849b594e696a6e8c1cedf" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:07.105449 13187 raft_consensus.cc:2454] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:07.105597 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61b85f4b849b594e696a6e8c1cedf" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:07.105736 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-9c5329f38f3343baaf43265990d86938 received from {username='slave'} at 127.6.196.197:53727
I20240502 14:15:07.105780 16020 raft_consensus.cc:2454] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:07.105847 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-9c5329f38f3343baaf43265990d86938 on tablet 9c5329f38f3343baaf43265990d86938 with peer d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:07.106670 16088 leader_election.cc:302] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:07.107152 19021 raft_consensus.cc:2781] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:07.107264 19021 raft_consensus.cc:482] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:07.107367 19021 raft_consensus.cc:3037] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.109277 19021 raft_consensus.cc:504] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:07.109792 19021 leader_election.cc:288] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:07.110965 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61b85f4b849b594e696a6e8c1cedf" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:07.111171 16020 raft_consensus.cc:3037] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.113201 16020 raft_consensus.cc:2454] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:07.113879 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97a61b85f4b849b594e696a6e8c1cedf" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:07.114058 13187 raft_consensus.cc:3037] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.114125 16090 leader_election.cc:302] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:07.114332 19021 raft_consensus.cc:2781] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:07.114485 19021 raft_consensus.cc:686] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:07.114706 19021 consensus_queue.cc:237] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:07.117695 13187 raft_consensus.cc:2454] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:07.135701 19100 ts_tablet_manager.cc:902] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:07.136512 13043 catalog_manager.cc:5484] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:07.141340 19090 tablet_copy_client.cc:308] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:36687
I20240502 14:15:07.142004 13208 tablet_copy_service.cc:140] P 20a967bad5504b1e9c6fb720ecdd1625: Received BeginTabletCopySession request for tablet b2905b7940fb45c49ef8b18943dae256 from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:37725)
I20240502 14:15:07.142171 13208 tablet_copy_service.cc:161] P 20a967bad5504b1e9c6fb720ecdd1625: Beginning new tablet copy session on tablet b2905b7940fb45c49ef8b18943dae256 from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:37725: session id = 8b47d4390f8343a493d6c00123785968-b2905b7940fb45c49ef8b18943dae256
I20240502 14:15:07.145090 13208 tablet_copy_source_session.cc:205] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:07.146437 19090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2905b7940fb45c49ef8b18943dae256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.146622 19100 tablet_copy_client.cc:308] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.199:33857
I20240502 14:15:07.147352 16308 tablet_copy_service.cc:140] P 5297e816f782430d9a01220f0471a882: Received BeginTabletCopySession request for tablet 7da975869e82479b91fdda30680e6b3c from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:55953)
I20240502 14:15:07.147488 16308 tablet_copy_service.cc:161] P 5297e816f782430d9a01220f0471a882: Beginning new tablet copy session on tablet 7da975869e82479b91fdda30680e6b3c from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:55953: session id = 20a967bad5504b1e9c6fb720ecdd1625-7da975869e82479b91fdda30680e6b3c
I20240502 14:15:07.149497 16308 tablet_copy_source_session.cc:205] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:07.152429 13444 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:15:07.152720 19090 tablet_copy_client.cc:783] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:07.152967 19090 tablet_copy_client.cc:647] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:07.153029 19021 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:07.150990 19100 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7da975869e82479b91fdda30680e6b3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.155203 19090 tablet_copy_client.cc:523] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:07.159848 19090 tablet_bootstrap.cc:492] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:07.163847 19100 tablet_copy_client.cc:783] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:07.164129 19100 tablet_copy_client.cc:647] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:07.166236 19100 tablet_copy_client.cc:523] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:07.175256 19100 tablet_bootstrap.cc:492] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:07.177575 19111 ts_tablet_manager.cc:902] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:07.185864 19100 tablet_bootstrap.cc:492] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:07.186657 19100 tablet_bootstrap.cc:492] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:15:07.186961 19100 ts_tablet_manager.cc:1367] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240502 14:15:07.187960 19100 raft_consensus.cc:348] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:07.188426 19100 raft_consensus.cc:729] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:15:07.188777 19100 consensus_queue.cc:260] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:07.198359 19100 ts_tablet_manager.cc:1398] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240502 14:15:07.202900 16308 tablet_copy_service.cc:342] P 5297e816f782430d9a01220f0471a882: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-7da975869e82479b91fdda30680e6b3c received from {username='slave'} at 127.6.196.193:55953
I20240502 14:15:07.203045 16308 tablet_copy_service.cc:433] P 5297e816f782430d9a01220f0471a882: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-7da975869e82479b91fdda30680e6b3c on tablet 7da975869e82479b91fdda30680e6b3c with peer 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:15:07.209916 13319 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:15:07.210718 19021 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:07.240466 13579 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:15:07.243033 18885 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:07.260578 13444 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:15:07.261457 19021 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:07.263724 19090 tablet_bootstrap.cc:492] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:07.264401 19090 tablet_bootstrap.cc:492] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:15:07.264822 19090 ts_tablet_manager.cc:1367] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.105s user 0.064s sys 0.017s
I20240502 14:15:07.269270 19090 raft_consensus.cc:348] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:07.269747 19090 raft_consensus.cc:729] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:15:07.270103 19090 consensus_queue.cc:260] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:07.276866 19090 ts_tablet_manager.cc:1398] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20240502 14:15:07.280025 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:07.286470 13208 tablet_copy_service.cc:342] P 20a967bad5504b1e9c6fb720ecdd1625: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-b2905b7940fb45c49ef8b18943dae256 received from {username='slave'} at 127.6.196.200:37725
I20240502 14:15:07.286618 13208 tablet_copy_service.cc:433] P 20a967bad5504b1e9c6fb720ecdd1625: ending tablet copy session 8b47d4390f8343a493d6c00123785968-b2905b7940fb45c49ef8b18943dae256 on tablet b2905b7940fb45c49ef8b18943dae256 with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:15:07.288403 13188 raft_consensus.cc:1261] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:07.289376 18885 consensus_queue.cc:1035] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.291352 16020 raft_consensus.cc:1261] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:07.292632 18885 consensus_queue.cc:1035] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.301862 13580 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:15:07.302476 19021 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:07.317303 13188 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:15:07.327855 19020 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:07.355926 18798 consensus_queue.cc:1035] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:15:07.369887 13121 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.198:42041, user_credentials={real_user=slave}} blocked reactor thread for 140817us
I20240502 14:15:07.377365 13043 catalog_manager.cc:5576] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695) does not have the latest schema for tablet 97a61b85f4b849b594e696a6e8c1cedf (table rebalancer_dynamic_table [id=9074beaa768c472299f9c09e6f7389bc]). Expected version 1 got 0
I20240502 14:15:07.381078 19072 ts_tablet_manager.cc:902] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:07.381876 19138 tablet.cc:1722] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:07.382649 19138 tablet.cc:1620] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:15:07.385341 13043 catalog_manager.cc:5576] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695) does not have the latest schema for tablet 97a61b85f4b849b594e696a6e8c1cedf (table rebalancer_dynamic_table [id=9074beaa768c472299f9c09e6f7389bc]). Expected version 1 got 0
I20240502 14:15:07.391431 18798 consensus_queue.cc:1035] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.413070 19072 tablet_copy_client.cc:308] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:15:07.413993 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet 652e49046212458280095c016a342f47 from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:40701)
I20240502 14:15:07.414165 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet 652e49046212458280095c016a342f47 from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:40701: session id = d1da7f11f0604f78934354a992bb0f55-652e49046212458280095c016a342f47
I20240502 14:15:07.417021 19119 ts_tablet_manager.cc:902] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:07.417297 13601 tablet_copy_source_session.cc:205] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:07.418380 19072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 652e49046212458280095c016a342f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.418475 19137 tablet.cc:1722] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:07.418927 19137 tablet.cc:1620] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:07.424706 19072 tablet_copy_client.cc:783] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:07.424969 19072 tablet_copy_client.cc:647] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:07.426411 19072 tablet_copy_client.cc:523] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:07.429525 13043 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=9074beaa768c472299f9c09e6f7389bc] alter complete (version 1)
I20240502 14:15:07.429675 19072 tablet_bootstrap.cc:492] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:07.450273 18175 consensus_queue.cc:1035] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.467348 13042 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet 97a61b85f4b849b594e696a6e8c1cedf (table rebalancer_dynamic_table [id=9074beaa768c472299f9c09e6f7389bc]). Expected version 1 got 0
I20240502 14:15:07.492940 18175 consensus_queue.cc:1035] T 6931796f283a4f20b7fec8c3163b8444 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.495918 19111 tablet_copy_client.cc:308] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:15:07.496683 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet d5975e02205a43d5b93ed8ab51f8a0f8 from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:56671)
I20240502 14:15:07.496822 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet d5975e02205a43d5b93ed8ab51f8a0f8 from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:56671: session id = 8b47d4390f8343a493d6c00123785968-d5975e02205a43d5b93ed8ab51f8a0f8
I20240502 14:15:07.500320 13339 tablet_copy_source_session.cc:205] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:07.501556 18668 consensus_queue.cc:1035] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.502902 19152 tablet.cc:1722] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:07.503522 19152 tablet.cc:1620] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:15:07.515966 19111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d5975e02205a43d5b93ed8ab51f8a0f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.543534 13043 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:07.553303 18668 consensus_queue.cc:1035] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:07.559146 19138 tablet.cc:1722] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:07.559685 19138 tablet.cc:1620] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:15:07.560468 19137 tablet.cc:1722] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:07.561605 19137 tablet.cc:1620] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:07.575306 13043 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=9074beaa768c472299f9c09e6f7389bc] alter complete (version 2)
I20240502 14:15:07.588691 19153 ts_tablet_manager.cc:902] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
W20240502 14:15:07.601267 13384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.197:43695, user_credentials={real_user=slave}} blocked reactor thread for 108569us
I20240502 14:15:07.600128 19152 tablet.cc:1722] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:07.601696 19111 tablet_copy_client.cc:783] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:07.601986 19111 tablet_copy_client.cc:647] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:07.602010 16022 raft_consensus.cc:1203] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:07.617817 19111 tablet_copy_client.cc:523] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:07.632844 16419 raft_consensus.cc:1203] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:15:07.634670 13045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:07.652112 19111 tablet_bootstrap.cc:492] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:07.654352 16401 tablet_service.cc:1467] Processing CreateTablet for tablet 90354ba8ac7f40a7a7428901eeda26bd (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cab90f7f1c84cf2b82ee344af1b6617]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.654773 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90354ba8ac7f40a7a7428901eeda26bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.660934 19152 tablet.cc:1620] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:15:07.662493 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 90354ba8ac7f40a7a7428901eeda26bd (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cab90f7f1c84cf2b82ee344af1b6617]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.662909 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90354ba8ac7f40a7a7428901eeda26bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.663391 16134 tablet_service.cc:1467] Processing CreateTablet for tablet db2bc203d47c4ca1b3329745a0ff7fe1 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cab90f7f1c84cf2b82ee344af1b6617]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.663749 16134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2bc203d47c4ca1b3329745a0ff7fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.664374 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 90354ba8ac7f40a7a7428901eeda26bd (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cab90f7f1c84cf2b82ee344af1b6617]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.664716 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90354ba8ac7f40a7a7428901eeda26bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.667841 16155 consensus_queue.cc:237] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:07.670679 13318 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:15:07.670992 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 154fe82369ef4d2bb38e8f14637463bb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cab90f7f1c84cf2b82ee344af1b6617]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.671183 19020 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:07.671382 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154fe82369ef4d2bb38e8f14637463bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.673518 19020 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:07.676311 16133 tablet_service.cc:1467] Processing CreateTablet for tablet 154fe82369ef4d2bb38e8f14637463bb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cab90f7f1c84cf2b82ee344af1b6617]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.676739 16133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154fe82369ef4d2bb38e8f14637463bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.679247 13318 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:07.683413 13427 tablet_service.cc:1467] Processing CreateTablet for tablet db2bc203d47c4ca1b3329745a0ff7fe1 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cab90f7f1c84cf2b82ee344af1b6617]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.683859 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2bc203d47c4ca1b3329745a0ff7fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.684053 13045 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:07.690797 13579 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:15:07.695540 19178 tablet_bootstrap.cc:492] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:07.697202 15999 tablet_service.cc:1467] Processing CreateTablet for tablet db2bc203d47c4ca1b3329745a0ff7fe1 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cab90f7f1c84cf2b82ee344af1b6617]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.697675 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2bc203d47c4ca1b3329745a0ff7fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.697842 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 154fe82369ef4d2bb38e8f14637463bb (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=6cab90f7f1c84cf2b82ee344af1b6617]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:07.698156 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154fe82369ef4d2bb38e8f14637463bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.705467 19178 tablet_bootstrap.cc:654] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.716667 19153 tablet_copy_client.cc:308] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:15:07.717409 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 1d607f6e65194399844bc14457fe8f15 from peer 88b77f6edc1c4a72b548bf47c05704f3 ({username='slave'} at 127.6.196.195:34735)
I20240502 14:15:07.717566 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 1d607f6e65194399844bc14457fe8f15 from peer 88b77f6edc1c4a72b548bf47c05704f3 at {username='slave'} at 127.6.196.195:34735: session id = 88b77f6edc1c4a72b548bf47c05704f3-1d607f6e65194399844bc14457fe8f15
I20240502 14:15:07.719594 19178 tablet_bootstrap.cc:492] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:07.719745 19178 ts_tablet_manager.cc:1367] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.006s
I20240502 14:15:07.720371 13339 tablet_copy_source_session.cc:205] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:07.720782 19178 raft_consensus.cc:348] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:07.721360 19178 raft_consensus.cc:374] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.721513 19178 raft_consensus.cc:729] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.721609 19153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d607f6e65194399844bc14457fe8f15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.722103 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e393a7736b14d5483973f9cb89058f8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:07.722088 19178 consensus_queue.cc:260] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:07.722657 19178 ts_tablet_manager.cc:1398] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:07.727412 19184 tablet_bootstrap.cc:492] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:07.728794 19184 tablet_bootstrap.cc:654] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.732167 13188 raft_consensus.cc:1203] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:07.734158 13046 catalog_manager.cc:5484] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 5 to 7, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 2 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:07.736343 19178 tablet_bootstrap.cc:492] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:07.729199 19180 tablet_bootstrap.cc:492] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:07.745761 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e393a7736b14d5483973f9cb89058f8 with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:07.747740 19180 tablet_bootstrap.cc:654] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.749002 19186 tablet_bootstrap.cc:492] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:07.748173 19020 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.020s
I20240502 14:15:07.750735 13580 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:07.751706 19186 tablet_bootstrap.cc:654] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.752404 19178 tablet_bootstrap.cc:654] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.755276 18175 raft_consensus.cc:482] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:07.755396 18175 raft_consensus.cc:504] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:07.756109 18175 leader_election.cc:288] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:07.756955 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90354ba8ac7f40a7a7428901eeda26bd" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
W20240502 14:15:07.760744 16355 leader_election.cc:341] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:07.763502 16151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90354ba8ac7f40a7a7428901eeda26bd" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
W20240502 14:15:07.764011 16354 leader_election.cc:341] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:07.764127 16354 leader_election.cc:302] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b47d4390f8343a493d6c00123785968; no voters: c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:07.765028 18175 raft_consensus.cc:2726] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:07.773716 19179 tablet_bootstrap.cc:492] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:07.775449 19179 tablet_bootstrap.cc:654] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.776351 19138 tablet.cc:1722] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:07.781220 19119 tablet_copy_client.cc:308] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:15:07.782456 19192 tablet_replica.cc:330] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:07.782665 19192 raft_consensus.cc:2229] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:07.782866 19192 raft_consensus.cc:2258] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:07.783403 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 6395ce4c30eb4644b0a711de3e60b9c0 from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:40385)
I20240502 14:15:07.783578 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 6395ce4c30eb4644b0a711de3e60b9c0 from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:40385: session id = c86f7440570f4796babfd44c3d7fd847-6395ce4c30eb4644b0a711de3e60b9c0
I20240502 14:15:07.783836 19192 ts_tablet_manager.cc:1875] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:07.786595 13339 tablet_copy_source_session.cc:205] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:07.787916 19184 tablet_bootstrap.cc:492] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:07.788334 19184 ts_tablet_manager.cc:1367] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.000s
I20240502 14:15:07.788959 19184 raft_consensus.cc:348] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:07.794256 19180 tablet_bootstrap.cc:492] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:07.794520 19180 ts_tablet_manager.cc:1367] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.000s
I20240502 14:15:07.795306 19137 tablet.cc:1722] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:07.795583 19180 raft_consensus.cc:348] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:07.795960 19180 raft_consensus.cc:374] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.796105 19180 raft_consensus.cc:729] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.796356 19184 raft_consensus.cc:374] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.796430 19184 raft_consensus.cc:729] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.796463 19180 consensus_queue.cc:260] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:07.796617 19184 consensus_queue.cc:260] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:07.797078 19184 ts_tablet_manager.cc:1398] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:15:07.797200 19180 ts_tablet_manager.cc:1398] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:07.797683 13043 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=9074beaa768c472299f9c09e6f7389bc] alter complete (version 3)
I20240502 14:15:07.800263 13044 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet 97a61b85f4b849b594e696a6e8c1cedf (table rebalancer_dynamic_table_new_name [id=9074beaa768c472299f9c09e6f7389bc]). Expected version 3 got 2
I20240502 14:15:07.801748 19179 tablet_bootstrap.cc:492] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:07.801868 19179 ts_tablet_manager.cc:1367] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240502 14:15:07.802563 19179 raft_consensus.cc:348] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:07.804255 19179 raft_consensus.cc:374] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.804442 19179 raft_consensus.cc:729] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.804733 19179 consensus_queue.cc:260] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:07.805517 19179 ts_tablet_manager.cc:1398] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:07.805913 19179 tablet_bootstrap.cc:492] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:07.807992 16151 consensus_queue.cc:237] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:07.810992 13580 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:15:07.811839 13188 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:15:07.812091 19021 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:15:07.813617 19021 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:15:07.815299 19021 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P c86f7440570f4796babfd44c3d7fd847 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:07.817510 19179 tablet_bootstrap.cc:654] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.819227 19119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6395ce4c30eb4644b0a711de3e60b9c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:07.819535 13580 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:07.823529 19152 tablet.cc:1722] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:07.824379 19192 ts_tablet_manager.cc:1888] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:15:07.824491 19192 log.cc:1196] T 0e393a7736b14d5483973f9cb89058f8 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/0e393a7736b14d5483973f9cb89058f8
I20240502 14:15:07.824482 13186 raft_consensus.cc:2932] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:07.827652 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7852539de07744caaf385620c5166861 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:07.831702 13043 catalog_manager.cc:5484] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 8 to 10, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 2 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:07.834017 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 0e393a7736b14d5483973f9cb89058f8 (table rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]) successfully deleted
I20240502 14:15:07.839845 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 7852539de07744caaf385620c5166861 with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:07.842532 19192 tablet_replica.cc:330] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:07.842762 19192 raft_consensus.cc:2229] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:07.842969 19192 raft_consensus.cc:2258] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:07.843938 19192 ts_tablet_manager.cc:1875] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:07.849573 19153 tablet_copy_client.cc:783] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:07.850409 19192 ts_tablet_manager.cc:1888] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 14:15:07.854368 19179 tablet_bootstrap.cc:492] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:07.854610 19179 ts_tablet_manager.cc:1367] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.000s
I20240502 14:15:07.855645 19179 raft_consensus.cc:348] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:07.856107 19179 raft_consensus.cc:374] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.856259 19179 raft_consensus.cc:729] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.856644 19179 consensus_queue.cc:260] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:07.857367 19179 ts_tablet_manager.cc:1398] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:07.858906 19179 tablet_bootstrap.cc:492] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:07.864097 19179 tablet_bootstrap.cc:654] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.883416 19179 tablet_bootstrap.cc:492] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:07.883911 19179 ts_tablet_manager.cc:1367] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20240502 14:15:07.850571 19153 tablet_copy_client.cc:647] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:07.885490 19192 log.cc:1196] T 7852539de07744caaf385620c5166861 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/7852539de07744caaf385620c5166861
I20240502 14:15:07.886185 19178 tablet_bootstrap.cc:492] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:07.886451 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 7852539de07744caaf385620c5166861 (table rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]) successfully deleted
I20240502 14:15:07.886333 19178 ts_tablet_manager.cc:1367] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.150s user 0.003s sys 0.002s
I20240502 14:15:07.887125 19153 tablet_copy_client.cc:523] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:07.887497 19178 raft_consensus.cc:348] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:07.887670 19178 raft_consensus.cc:374] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.887717 19178 raft_consensus.cc:729] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.887984 19178 consensus_queue.cc:260] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:07.888571 19178 ts_tablet_manager.cc:1398] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:07.890553 19153 tablet_bootstrap.cc:492] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:07.890615 19179 raft_consensus.cc:348] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:07.890798 19179 raft_consensus.cc:374] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.892055 19179 raft_consensus.cc:729] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.893695 19179 consensus_queue.cc:260] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:07.898483 19179 ts_tablet_manager.cc:1398] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 14:15:07.898701 19186 tablet_bootstrap.cc:492] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:07.898842 19186 ts_tablet_manager.cc:1367] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.150s user 0.015s sys 0.075s
I20240502 14:15:07.899538 19186 raft_consensus.cc:348] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:07.899727 19186 raft_consensus.cc:374] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.899803 19186 raft_consensus.cc:729] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.900041 19186 consensus_queue.cc:260] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:07.900586 19186 ts_tablet_manager.cc:1398] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:07.900846 19186 tablet_bootstrap.cc:492] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:07.902379 19186 tablet_bootstrap.cc:654] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:07.903776 19119 tablet_copy_client.cc:783] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:07.904037 19119 tablet_copy_client.cc:647] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:07.905593 19119 tablet_copy_client.cc:523] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:07.906719 13043 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:07.906912 13043 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:07.909777 13043 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 97a61b85f4b849b594e696a6e8c1cedf
I20240502 14:15:07.910521 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a61b85f4b849b594e696a6e8c1cedf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:07 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:07.911561 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a61b85f4b849b594e696a6e8c1cedf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:07 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:15:07.912495 15999 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a61b85f4b849b594e696a6e8c1cedf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:07 UTC) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:07.917002 19186 tablet_bootstrap.cc:492] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:07.917165 19186 ts_tablet_manager.cc:1367] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240502 14:15:07.917583 19207 tablet_replica.cc:330] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:07.917758 19207 raft_consensus.cc:2229] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:07.917883 19186 raft_consensus.cc:348] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:07.918064 19186 raft_consensus.cc:374] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:07.918143 19207 raft_consensus.cc:2258] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:07.918155 19186 raft_consensus.cc:729] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:07.918388 19186 consensus_queue.cc:260] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:07.919103 19186 ts_tablet_manager.cc:1398] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:07.921427 19119 tablet_bootstrap.cc:492] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:07.924435 19207 ts_tablet_manager.cc:1875] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:07.935086 19207 ts_tablet_manager.cc:1888] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:07.935204 19207 log.cc:1196] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/97a61b85f4b849b594e696a6e8c1cedf
I20240502 14:15:07.935737 19207 ts_tablet_manager.cc:1909] T 97a61b85f4b849b594e696a6e8c1cedf P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:15:07.937510 19209 tablet_replica.cc:330] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:07.937814 19209 raft_consensus.cc:2229] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:07.938246 18668 raft_consensus.cc:482] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:07.938416 18668 raft_consensus.cc:504] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:07.939014 18668 leader_election.cc:288] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:07.939962 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154fe82369ef4d2bb38e8f14637463bb" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:07.940171 16418 raft_consensus.cc:2454] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:07.940438 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 97a61b85f4b849b594e696a6e8c1cedf (table rebalancer_dynamic_table_new_name [id=9074beaa768c472299f9c09e6f7389bc]) successfully deleted
I20240502 14:15:07.940764 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154fe82369ef4d2bb38e8f14637463bb" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:07.940937 16152 raft_consensus.cc:2454] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:07.941475 15956 leader_election.cc:302] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:07.942057 18668 raft_consensus.cc:2781] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:07.942157 18668 raft_consensus.cc:482] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:07.942252 18668 raft_consensus.cc:3037] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.944689 18668 raft_consensus.cc:504] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:07.946266 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154fe82369ef4d2bb38e8f14637463bb" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:07.946439 16418 raft_consensus.cc:3037] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.948868 16418 raft_consensus.cc:2454] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:07.949335 15956 leader_election.cc:302] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:07.949818 19171 raft_consensus.cc:2781] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:07.949956 19171 raft_consensus.cc:686] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:07.950237 19171 consensus_queue.cc:237] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:07.953915 18668 leader_election.cc:288] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:07.954500 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "154fe82369ef4d2bb38e8f14637463bb" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:07.954695 16152 raft_consensus.cc:3037] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.956750 16152 raft_consensus.cc:2454] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:07.956022 19072 tablet_bootstrap.cc:492] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:07.957653 19072 tablet_bootstrap.cc:492] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:15:07.958062 19072 ts_tablet_manager.cc:1367] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.528s user 0.083s sys 0.093s
I20240502 14:15:07.959022 19072 raft_consensus.cc:348] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:07.959311 19072 raft_consensus.cc:729] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:15:07.959533 19072 consensus_queue.cc:260] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:07.960296 19072 ts_tablet_manager.cc:1398] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:07.960306 19021 raft_consensus.cc:482] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:07.960448 19021 raft_consensus.cc:504] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:07.961033 19021 leader_election.cc:288] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:07.961493 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2bc203d47c4ca1b3329745a0ff7fe1" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:07.961680 13447 raft_consensus.cc:2454] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:07.962062 16088 leader_election.cc:302] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:07.962478 19021 raft_consensus.cc:2781] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:07.962558 19021 raft_consensus.cc:482] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:07.962638 19021 raft_consensus.cc:3037] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.964840 19021 raft_consensus.cc:504] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:07.965416 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2bc203d47c4ca1b3329745a0ff7fe1" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:07.965623 16020 raft_consensus.cc:2454] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:07.967901 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2bc203d47c4ca1b3329745a0ff7fe1" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:07.968055 16020 raft_consensus.cc:3037] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.970427 16020 raft_consensus.cc:2454] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:07.970957 16090 leader_election.cc:302] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:07.971117 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-652e49046212458280095c016a342f47 received from {username='slave'} at 127.6.196.197:40701
I20240502 14:15:07.971227 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-652e49046212458280095c016a342f47 on tablet 652e49046212458280095c016a342f47 with peer d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:07.971258 19020 raft_consensus.cc:2781] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:07.971362 19020 raft_consensus.cc:686] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:07.971607 19020 consensus_queue.cc:237] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:07.972630 19208 tablet_replica.cc:330] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:15:07.972937 19208 raft_consensus.cc:2229] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:07.973191 19208 raft_consensus.cc:2258] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:07.973623 19021 leader_election.cc:288] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:07.973929 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2bc203d47c4ca1b3329745a0ff7fe1" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:07.974119 19208 ts_tablet_manager.cc:1875] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:07.974144 13447 raft_consensus.cc:3037] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:07.976477 19209 raft_consensus.cc:2258] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:07.979023 13046 catalog_manager.cc:5484] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:07.982324 13447 raft_consensus.cc:2454] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:07.984265 19209 ts_tablet_manager.cc:1875] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:07.986935 13046 catalog_manager.cc:5484] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:07.991768 19209 ts_tablet_manager.cc:1888] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:07.991945 19209 log.cc:1196] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/97a61b85f4b849b594e696a6e8c1cedf
I20240502 14:15:07.992327 19153 tablet_bootstrap.cc:492] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:07.992743 19209 ts_tablet_manager.cc:1909] T 97a61b85f4b849b594e696a6e8c1cedf P d1da7f11f0604f78934354a992bb0f55: Deleting consensus metadata
I20240502 14:15:07.992774 19153 tablet_bootstrap.cc:492] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap complete.
I20240502 14:15:07.993026 19153 ts_tablet_manager.cc:1367] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.103s user 0.006s sys 0.005s
I20240502 14:15:07.993958 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet 97a61b85f4b849b594e696a6e8c1cedf (table rebalancer_dynamic_table_new_name [id=9074beaa768c472299f9c09e6f7389bc]) successfully deleted
I20240502 14:15:07.997920 19153 raft_consensus.cc:348] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:07.998132 19153 raft_consensus.cc:729] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: LEARNER
I20240502 14:15:07.998667 19153 consensus_queue.cc:260] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:08.005611 19153 ts_tablet_manager.cc:1398] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.012s user 0.000s sys 0.001s
I20240502 14:15:08.013355 19208 ts_tablet_manager.cc:1888] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:08.013906 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:08.016029 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-1d607f6e65194399844bc14457fe8f15 received from {username='slave'} at 127.6.196.195:34735
I20240502 14:15:08.016165 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-1d607f6e65194399844bc14457fe8f15 on tablet 1d607f6e65194399844bc14457fe8f15 with peer 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:15:08.021765 19208 log.cc:1196] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/97a61b85f4b849b594e696a6e8c1cedf
I20240502 14:15:08.026429 19208 ts_tablet_manager.cc:1909] T 97a61b85f4b849b594e696a6e8c1cedf P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:15:08.030396 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 22be14aaab2f4ac89baf1bf58bccc8f1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0db2138ef18044cf9d18f9c87597b57e]), partition=
I20240502 14:15:08.030706 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22be14aaab2f4ac89baf1bf58bccc8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.032979 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 97a61b85f4b849b594e696a6e8c1cedf (table rebalancer_dynamic_table_new_name [id=9074beaa768c472299f9c09e6f7389bc]) successfully deleted
I20240502 14:15:08.033932 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 22be14aaab2f4ac89baf1bf58bccc8f1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0db2138ef18044cf9d18f9c87597b57e]), partition=
I20240502 14:15:08.034271 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22be14aaab2f4ac89baf1bf58bccc8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.037703 19214 tablet_bootstrap.cc:492] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:08.038473 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 22be14aaab2f4ac89baf1bf58bccc8f1 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0db2138ef18044cf9d18f9c87597b57e]), partition=
I20240502 14:15:08.038779 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22be14aaab2f4ac89baf1bf58bccc8f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.039134 19214 tablet_bootstrap.cc:654] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.043483 19179 tablet_bootstrap.cc:492] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:08.044934 19179 tablet_bootstrap.cc:654] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.050611 19119 tablet_bootstrap.cc:492] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:08.051115 19119 tablet_bootstrap.cc:492] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:15:08.051374 19119 ts_tablet_manager.cc:1367] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.130s user 0.045s sys 0.000s
I20240502 14:15:08.052557 19119 raft_consensus.cc:348] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:08.052798 19119 raft_consensus.cc:729] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:15:08.053452 19119 consensus_queue.cc:260] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:08.053730 19214 tablet_bootstrap.cc:492] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:08.054009 19214 ts_tablet_manager.cc:1367] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.005s
I20240502 14:15:08.054708 19214 raft_consensus.cc:348] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.054952 19214 raft_consensus.cc:374] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.055051 19214 raft_consensus.cc:729] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.055334 19214 consensus_queue.cc:260] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.055938 19214 ts_tablet_manager.cc:1398] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:08.057924 19119 ts_tablet_manager.cc:1398] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:15:08.058626 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-6395ce4c30eb4644b0a711de3e60b9c0 received from {username='slave'} at 127.6.196.198:40385
I20240502 14:15:08.058744 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-6395ce4c30eb4644b0a711de3e60b9c0 on tablet 6395ce4c30eb4644b0a711de3e60b9c0 with peer c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:08.069728 19217 tablet_bootstrap.cc:492] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:08.071604 19179 tablet_bootstrap.cc:492] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:08.071811 19179 ts_tablet_manager.cc:1367] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.011s
I20240502 14:15:08.072355 19179 raft_consensus.cc:348] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.072517 19179 raft_consensus.cc:374] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.072597 19179 raft_consensus.cc:729] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.072806 19179 consensus_queue.cc:260] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.075538 19179 ts_tablet_manager.cc:1398] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:15:08.077510 19217 tablet_bootstrap.cc:654] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.077766 19020 raft_consensus.cc:482] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:08.077931 19020 raft_consensus.cc:504] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:08.078593 19020 leader_election.cc:288] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:08.079078 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90354ba8ac7f40a7a7428901eeda26bd" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:08.079294 16418 raft_consensus.cc:2454] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:08.079574 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90354ba8ac7f40a7a7428901eeda26bd" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:08.079756 13319 raft_consensus.cc:2454] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:08.080291 16089 leader_election.cc:302] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:08.080765 19020 raft_consensus.cc:2781] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:08.080860 19020 raft_consensus.cc:482] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:08.080968 19020 raft_consensus.cc:3037] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.088757 19217 tablet_bootstrap.cc:492] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:08.089077 19217 ts_tablet_manager.cc:1367] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.004s
I20240502 14:15:08.089824 19217 raft_consensus.cc:348] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.090142 19217 raft_consensus.cc:374] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.090281 19217 raft_consensus.cc:729] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.090572 19217 consensus_queue.cc:260] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.091261 19217 ts_tablet_manager.cc:1398] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:08.084733 19020 raft_consensus.cc:504] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:08.096585 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90354ba8ac7f40a7a7428901eeda26bd" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:08.096784 16418 raft_consensus.cc:3037] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.099601 16418 raft_consensus.cc:2454] T 90354ba8ac7f40a7a7428901eeda26bd P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:08.100023 16089 leader_election.cc:302] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:08.100329 19021 raft_consensus.cc:2781] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:08.100486 19021 raft_consensus.cc:686] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:08.100728 19021 consensus_queue.cc:237] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:08.102766 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90354ba8ac7f40a7a7428901eeda26bd" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:08.102948 13318 raft_consensus.cc:3037] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.107806 13318 raft_consensus.cc:2454] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:08.108238 19020 leader_election.cc:288] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:08.111610 19111 tablet_bootstrap.cc:492] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:08.112063 19111 tablet_bootstrap.cc:492] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:15:08.112381 19111 ts_tablet_manager.cc:1367] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.460s user 0.042s sys 0.005s
I20240502 14:15:08.113116 19111 raft_consensus.cc:348] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:08.113297 16152 raft_consensus.cc:1203] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:08.113374 19111 raft_consensus.cc:729] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:15:08.113564 19111 consensus_queue.cc:260] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:08.114028 19111 ts_tablet_manager.cc:1398] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240502 14:15:08.114883 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-d5975e02205a43d5b93ed8ab51f8a0f8 received from {username='slave'} at 127.6.196.200:56671
I20240502 14:15:08.114987 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 8b47d4390f8343a493d6c00123785968-d5975e02205a43d5b93ed8ab51f8a0f8 on tablet d5975e02205a43d5b93ed8ab51f8a0f8 with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:15:08.116109 19020 raft_consensus.cc:482] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:08.116312 19020 raft_consensus.cc:504] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.117616 16285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22be14aaab2f4ac89baf1bf58bccc8f1" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:08.117820 16285 raft_consensus.cc:2454] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:08.118427 16090 leader_election.cc:302] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:08.119088 19020 leader_election.cc:288] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:08.119329 19020 raft_consensus.cc:2781] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:08.119452 19020 raft_consensus.cc:482] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:08.119603 19020 raft_consensus.cc:3037] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.120203 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22be14aaab2f4ac89baf1bf58bccc8f1" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:08.120407 13580 raft_consensus.cc:2454] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:08.122745 19020 raft_consensus.cc:504] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.123432 19020 leader_election.cc:288] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:08.124471 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22be14aaab2f4ac89baf1bf58bccc8f1" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:08.124627 13580 raft_consensus.cc:3037] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.124621 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22be14aaab2f4ac89baf1bf58bccc8f1" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:08.124811 16288 raft_consensus.cc:3037] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.126466 13580 raft_consensus.cc:2454] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:08.127379 16288 raft_consensus.cc:2454] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:08.127730 16090 leader_election.cc:302] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:08.127951 19020 raft_consensus.cc:2781] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:08.128099 19020 raft_consensus.cc:686] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:08.128294 19020 consensus_queue.cc:237] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.139279 13045 catalog_manager.cc:5484] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:08.139847 13045 catalog_manager.cc:5484] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:08.161895 19151 raft_consensus.cc:1050] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER d1da7f11f0604f78934354a992bb0f55 to VOTER
I20240502 14:15:08.162674 19151 consensus_queue.cc:237] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:08.174934 13188 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:08.175288 13318 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:08.175436 16020 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:08.176003 19155 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:08.176349 19151 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:08.176759 19154 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:08.182300 13447 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3d39f694c8554638a3b10ebf7cd01350"
dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51180
I20240502 14:15:08.182446 13447 raft_consensus.cc:595] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:08.193430 19154 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:08.197014 16020 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:08.200449 13044 catalog_manager.cc:5484] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:08.200719 13186 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:08.210721 18449 raft_consensus.cc:1050] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:15:08.211596 18449 consensus_queue.cc:237] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:08.212556 13447 consensus_queue.cc:237] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:08.214995 16020 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:08.215744 13186 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:08.216733 19259 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:08.217686 19151 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:08.218106 16418 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:08.219159 16022 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:08.219274 18449 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:08.219756 18449 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:08.219790 19154 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:08.220367 16020 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:08.224561 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9c5329f38f3343baaf43265990d86938 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:08.225072 13318 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:08.225569 19132 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:08.226939 13042 catalog_manager.cc:5484] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:08.231328 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5329f38f3343baaf43265990d86938 with delete_type TABLET_DATA_TOMBSTONED (TS cd67493f6f0142f896010f2b8a39887d not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:57492
I20240502 14:15:08.234977 19183 raft_consensus.cc:1050] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882: attempting to promote NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 to VOTER
I20240502 14:15:08.235766 19183 consensus_queue.cc:237] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:08.236479 13186 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:08.237085 19267 tablet_replica.cc:330] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:15:08.237294 19267 raft_consensus.cc:2229] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:08.238447 19267 pending_rounds.cc:62] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d: Trying to abort 1 pending ops.
I20240502 14:15:08.238497 19267 pending_rounds.cc:68] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7023244526232289280 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "9c5329f38f3343baaf43265990d86938" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } } }
I20240502 14:15:08.238786 19267 raft_consensus.cc:2894] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240502 14:15:08.238916 19267 raft_consensus.cc:2866] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:15:08.239003 19267 raft_consensus.cc:2258] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:08.240864 18449 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.241535 16020 raft_consensus.cc:1261] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:08.241786 13188 raft_consensus.cc:1261] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:08.242002 18788 consensus_queue.cc:1035] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:08.242460 18788 consensus_queue.cc:1035] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:08.246017 16419 raft_consensus.cc:1261] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:08.245741 16418 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.246642 19176 consensus_queue.cc:1035] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:08.252249 13045 catalog_manager.cc:5484] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:08.252875 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:08.255483 16022 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.240537 19267 ts_tablet_manager.cc:1875] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:08.260356 13580 raft_consensus.cc:1261] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:08.262427 16288 raft_consensus.cc:1261] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:08.264333 19021 consensus_queue.cc:1035] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.267256 19021 consensus_queue.cc:1035] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.267562 13186 consensus_queue.cc:237] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:08.270077 19267 ts_tablet_manager.cc:1888] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:08.273370 19259 raft_consensus.cc:979] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: : Instructing follower cd67493f6f0142f896010f2b8a39887d to start an election
I20240502 14:15:08.273478 19259 raft_consensus.cc:1067] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Signalling peer cd67493f6f0142f896010f2b8a39887d to start an election
I20240502 14:15:08.275372 16023 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:08.269112 13319 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.274160 13318 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3d39f694c8554638a3b10ebf7cd01350"
dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
from {username='slave'} at 127.6.196.195:34735
I20240502 14:15:08.275918 13318 raft_consensus.cc:482] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:08.276113 13318 raft_consensus.cc:3037] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:08.276355 16418 raft_consensus.cc:1261] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:08.276365 19273 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:08.277042 19262 consensus_queue.cc:1035] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:08.278828 13318 raft_consensus.cc:504] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:08.279597 16023 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.280737 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d39f694c8554638a3b10ebf7cd01350" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:08.280916 13447 raft_consensus.cc:3032] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:08.280974 13447 raft_consensus.cc:729] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:08.281217 13447 consensus_queue.cc:260] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:08.281589 13447 raft_consensus.cc:3037] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:08.281682 13318 leader_election.cc:288] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:08.282150 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d39f694c8554638a3b10ebf7cd01350" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:08.282320 16152 raft_consensus.cc:3037] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:08.284253 19267 log.cc:1196] T 9c5329f38f3343baaf43265990d86938 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/9c5329f38f3343baaf43265990d86938
I20240502 14:15:08.284799 16152 raft_consensus.cc:2454] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 2.
I20240502 14:15:08.285164 16420 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.285771 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 9c5329f38f3343baaf43265990d86938 (table rebalance_test_table_3 [id=8fa288258a0f4231be6de034088616ab]) successfully deleted
I20240502 14:15:08.288787 13447 raft_consensus.cc:2454] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 2.
I20240502 14:15:08.288961 19262 raft_consensus.cc:2932] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.289255 13251 leader_election.cc:302] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:08.290774 19022 raft_consensus.cc:2781] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:08.290922 19022 raft_consensus.cc:686] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 2 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:08.291213 19022 consensus_queue.cc:237] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:08.292636 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b2905b7940fb45c49ef8b18943dae256 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:08.294191 13042 catalog_manager.cc:5484] T b2905b7940fb45c49ef8b18943dae256 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 8 to 9, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.300518 16418 raft_consensus.cc:1203] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:15:08.301512 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d39f694c8554638a3b10ebf7cd01350" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:08.301769 13186 raft_consensus.cc:3037] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:08.303366 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet b2905b7940fb45c49ef8b18943dae256 with delete_type TABLET_DATA_TOMBSTONED (TS cd67493f6f0142f896010f2b8a39887d not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:57492
I20240502 14:15:08.303753 19267 tablet_replica.cc:330] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:15:08.304071 19267 raft_consensus.cc:2229] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:08.304402 19267 raft_consensus.cc:2258] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:08.314625 19267 ts_tablet_manager.cc:1875] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:08.319671 19183 raft_consensus.cc:2932] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:08.321182 19267 ts_tablet_manager.cc:1888] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:15:08.321398 19267 log.cc:1196] T b2905b7940fb45c49ef8b18943dae256 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/b2905b7940fb45c49ef8b18943dae256
I20240502 14:15:08.322168 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet b2905b7940fb45c49ef8b18943dae256 (table rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]) successfully deleted
I20240502 14:15:08.323545 13045 catalog_manager.cc:5484] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 1 to 2, leader changed from 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 2 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:08.324519 13186 raft_consensus.cc:2454] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 2.
I20240502 14:15:08.324728 16419 raft_consensus.cc:2932] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:08.333554 13046 catalog_manager.cc:5484] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:08.334122 16022 raft_consensus.cc:2932] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:08.336313 13186 raft_consensus.cc:2932] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:08.372623 19171 consensus_queue.cc:1035] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.384524 13045 catalog_manager.cc:5576] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817) does not have the latest schema for tablet 22be14aaab2f4ac89baf1bf58bccc8f1 (table rebalancer_dynamic_table [id=0db2138ef18044cf9d18f9c87597b57e]). Expected version 1 got 0
I20240502 14:15:08.390130 19021 consensus_queue.cc:1035] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.411087 13045 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet 22be14aaab2f4ac89baf1bf58bccc8f1 (table rebalancer_dynamic_table [id=0db2138ef18044cf9d18f9c87597b57e]). Expected version 1 got 0
I20240502 14:15:08.412191 19289 tablet.cc:1722] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:08.412699 19289 tablet.cc:1620] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: MemRowSet was empty: no flush needed.
I20240502 14:15:08.413345 13045 catalog_manager.cc:5576] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817) does not have the latest schema for tablet 22be14aaab2f4ac89baf1bf58bccc8f1 (table rebalancer_dynamic_table [id=0db2138ef18044cf9d18f9c87597b57e]). Expected version 1 got 0
I20240502 14:15:08.418423 13042 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet 22be14aaab2f4ac89baf1bf58bccc8f1 (table rebalancer_dynamic_table [id=0db2138ef18044cf9d18f9c87597b57e]). Expected version 1 got 0
I20240502 14:15:08.419514 19290 tablet.cc:1722] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:08.420202 19290 tablet.cc:1620] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:08.421527 19288 tablet.cc:1722] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:08.421963 19288 tablet.cc:1620] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:08.421964 13042 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 22be14aaab2f4ac89baf1bf58bccc8f1 (table rebalancer_dynamic_table [id=0db2138ef18044cf9d18f9c87597b57e]). Expected version 1 got 0
I20240502 14:15:08.427798 16022 raft_consensus.cc:1203] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:15:08.432921 13045 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=0db2138ef18044cf9d18f9c87597b57e] alter complete (version 1)
I20240502 14:15:08.439337 18668 consensus_queue.cc:1035] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.439592 19020 consensus_queue.cc:1035] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.489594 13444 raft_consensus.cc:1203] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:08.499930 13580 consensus_queue.cc:237] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:08.503311 16022 raft_consensus.cc:1261] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:08.503427 13186 raft_consensus.cc:1261] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:08.503996 13514 consensus_peers.cc:485] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e17e489fe5e04a758d9d9d0259159c64. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:08.504160 19313 consensus_queue.cc:1035] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:08.505079 19313 consensus_queue.cc:1035] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:08.508249 18881 raft_consensus.cc:2932] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:08.511034 13186 raft_consensus.cc:2932] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:08.511303 16022 raft_consensus.cc:2932] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:08.518909 13042 catalog_manager.cc:5484] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:08.540670 16022 consensus_queue.cc:237] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:08.542660 16419 raft_consensus.cc:1261] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:08.542668 16152 raft_consensus.cc:1261] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:08.543110 15955 consensus_peers.cc:485] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8a98395ed74c4c2eb265a141aec14bd3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:08.543265 19171 consensus_queue.cc:1035] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:08.543534 18668 consensus_queue.cc:1035] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:08.546221 19173 raft_consensus.cc:2932] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:08.547071 16152 raft_consensus.cc:2932] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:08.547173 16419 raft_consensus.cc:2932] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:08.555554 13046 catalog_manager.cc:5484] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:08.562678 13046 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:08.572743 19288 tablet.cc:1722] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:08.573251 19288 tablet.cc:1620] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:08.573419 19290 tablet.cc:1722] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:08.573851 19290 tablet.cc:1620] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:08.577040 19289 tablet.cc:1722] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:08.577497 19289 tablet.cc:1620] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: MemRowSet was empty: no flush needed.
I20240502 14:15:08.582921 13045 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=0db2138ef18044cf9d18f9c87597b57e] alter complete (version 2)
I20240502 14:15:08.614444 18885 consensus_queue.cc:1035] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.621980 19021 consensus_queue.cc:1035] T 90354ba8ac7f40a7a7428901eeda26bd P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:08.645498 16285 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7da975869e82479b91fdda30680e6b3c"
dest_uuid: "5297e816f782430d9a01220f0471a882"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:48874
I20240502 14:15:08.645781 16285 raft_consensus.cc:595] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:08.667702 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:08.679603 16401 tablet_service.cc:1467] Processing CreateTablet for tablet d7db78325b44450a87520a14a2444981 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=48aed9d8400b44738c091a427387612f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:08.680164 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7db78325b44450a87520a14a2444981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.680958 13427 tablet_service.cc:1467] Processing CreateTablet for tablet d7db78325b44450a87520a14a2444981 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=48aed9d8400b44738c091a427387612f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:08.681393 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7db78325b44450a87520a14a2444981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.685454 19373 tablet_bootstrap.cc:492] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:08.686303 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 76b46f71180f4269bee53f206e9c2665 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=48aed9d8400b44738c091a427387612f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:08.686846 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b46f71180f4269bee53f206e9c2665. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.687702 19373 tablet_bootstrap.cc:654] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.688730 16265 tablet_service.cc:1467] Processing CreateTablet for tablet caac1154570d4420ae8764ca18667f29 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=48aed9d8400b44738c091a427387612f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:08.689105 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caac1154570d4420ae8764ca18667f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.689030 15999 tablet_service.cc:1467] Processing CreateTablet for tablet caac1154570d4420ae8764ca18667f29 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=48aed9d8400b44738c091a427387612f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:08.689476 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caac1154570d4420ae8764ca18667f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.689535 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 76b46f71180f4269bee53f206e9c2665 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=48aed9d8400b44738c091a427387612f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:08.689657 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 76b46f71180f4269bee53f206e9c2665 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=48aed9d8400b44738c091a427387612f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:08.689836 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b46f71180f4269bee53f206e9c2665. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.690091 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b46f71180f4269bee53f206e9c2665. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.694367 13297 tablet_service.cc:1467] Processing CreateTablet for tablet d7db78325b44450a87520a14a2444981 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=48aed9d8400b44738c091a427387612f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:08.694839 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d7db78325b44450a87520a14a2444981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.696991 13046 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:08.697474 19023 raft_consensus.cc:1050] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER c86f7440570f4796babfd44c3d7fd847 to VOTER
I20240502 14:15:08.697844 13298 tablet_service.cc:1467] Processing CreateTablet for tablet caac1154570d4420ae8764ca18667f29 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=48aed9d8400b44738c091a427387612f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:08.698230 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caac1154570d4420ae8764ca18667f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.698321 19023 consensus_queue.cc:237] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:08.707441 19024 raft_consensus.cc:1050] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:15:08.708169 19024 consensus_queue.cc:237] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:08.710888 19288 tablet.cc:1722] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:08.711675 19290 tablet.cc:1722] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:08.715387 19289 tablet.cc:1722] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:08.722188 19376 tablet_bootstrap.cc:492] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:08.726766 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=0db2138ef18044cf9d18f9c87597b57e] alter complete (version 3)
I20240502 14:15:08.740964 19381 tablet_bootstrap.cc:492] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:08.741560 16419 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:08.742190 19022 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:08.742755 19381 tablet_bootstrap.cc:654] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.744107 19382 tablet_bootstrap.cc:492] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:08.745072 16020 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:08.745575 19022 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:08.745744 19382 tablet_bootstrap.cc:654] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.745779 19373 tablet_bootstrap.cc:492] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:08.745971 19373 ts_tablet_manager.cc:1367] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.061s user 0.008s sys 0.015s
I20240502 14:15:08.746850 19373 raft_consensus.cc:348] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:08.747136 19373 raft_consensus.cc:374] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.747203 19373 raft_consensus.cc:729] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.747455 19373 consensus_queue.cc:260] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:08.749421 19373 ts_tablet_manager.cc:1398] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:08.750562 16155 raft_consensus.cc:1261] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:08.751963 19376 tablet_bootstrap.cc:654] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.752103 13444 raft_consensus.cc:1261] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:08.752594 19022 consensus_queue.cc:1035] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:08.754235 19381 tablet_bootstrap.cc:492] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:08.756282 19382 tablet_bootstrap.cc:492] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:08.754050 19022 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.756659 19381 ts_tablet_manager.cc:1367] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240502 14:15:08.757699 19382 ts_tablet_manager.cc:1367] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240502 14:15:08.758172 13580 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:08.758628 19382 raft_consensus.cc:348] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.758816 19382 raft_consensus.cc:374] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.758914 19382 raft_consensus.cc:729] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.759120 19382 consensus_queue.cc:260] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.760239 19381 raft_consensus.cc:348] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:08.761371 19381 raft_consensus.cc:374] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.761600 19381 raft_consensus.cc:729] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.761979 19381 consensus_queue.cc:260] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:08.763079 19381 ts_tablet_manager.cc:1398] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:15:08.767676 19382 ts_tablet_manager.cc:1398] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:15:08.755761 19395 consensus_queue.cc:1035] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:08.773013 19395 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:08.768433 19381 tablet_bootstrap.cc:492] T d7db78325b44450a87520a14a2444981 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:08.775125 19382 tablet_bootstrap.cc:492] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:08.778834 19381 tablet_bootstrap.cc:654] T d7db78325b44450a87520a14a2444981 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.779815 19382 tablet_bootstrap.cc:654] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.780687 13186 raft_consensus.cc:1261] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:08.781951 13045 catalog_manager.cc:5484] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:08.783970 19022 consensus_queue.cc:1035] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:08.785094 19382 tablet_bootstrap.cc:492] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:08.785997 19382 ts_tablet_manager.cc:1367] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20240502 14:15:08.785097 16419 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.787248 19382 raft_consensus.cc:348] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:08.787446 19382 raft_consensus.cc:374] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.787513 19382 raft_consensus.cc:729] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.788554 19381 tablet_bootstrap.cc:492] T d7db78325b44450a87520a14a2444981 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:08.788679 19381 ts_tablet_manager.cc:1367] T d7db78325b44450a87520a14a2444981 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 14:15:08.790309 19382 consensus_queue.cc:260] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:08.790985 19381 raft_consensus.cc:348] T d7db78325b44450a87520a14a2444981 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:08.792692 16020 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.800845 19279 raft_consensus.cc:979] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882: : Instructing follower d1da7f11f0604f78934354a992bb0f55 to start an election
I20240502 14:15:08.800990 19279 raft_consensus.cc:1067] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Signalling peer d1da7f11f0604f78934354a992bb0f55 to start an election
I20240502 14:15:08.805114 16022 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7da975869e82479b91fdda30680e6b3c"
dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
from {username='slave'} at 127.6.196.199:60087
I20240502 14:15:08.805383 16022 raft_consensus.cc:482] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:08.805537 16022 raft_consensus.cc:3037] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:08.808328 16022 raft_consensus.cc:504] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:08.809576 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da975869e82479b91fdda30680e6b3c" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:08.809890 16419 raft_consensus.cc:3037] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:08.810416 16285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da975869e82479b91fdda30680e6b3c" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:08.810626 16285 raft_consensus.cc:3032] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:08.810753 16285 raft_consensus.cc:729] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:08.811123 16285 consensus_queue.cc:260] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:08.811604 16285 raft_consensus.cc:3037] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:08.814386 16285 raft_consensus.cc:2454] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 2.
I20240502 14:15:08.816991 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7da975869e82479b91fdda30680e6b3c" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:08.818645 16419 raft_consensus.cc:2454] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 2.
I20240502 14:15:08.819074 15956 leader_election.cc:302] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:08.819388 18668 raft_consensus.cc:2781] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:08.819546 18668 raft_consensus.cc:686] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 2 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:08.817207 13188 raft_consensus.cc:3037] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:08.819818 18668 consensus_queue.cc:237] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:08.822233 13188 raft_consensus.cc:2454] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 2.
I20240502 14:15:08.824478 13044 catalog_manager.cc:5484] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 1 to 2, leader changed from 5297e816f782430d9a01220f0471a882 (127.6.196.199) to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 2 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:08.824968 13185 raft_consensus.cc:1261] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:15:08.825829 16022 leader_election.cc:288] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:08.829777 13044 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:08.830051 13044 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:08.832998 13044 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 22be14aaab2f4ac89baf1bf58bccc8f1
I20240502 14:15:08.833694 19376 tablet_bootstrap.cc:492] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:08.833695 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet 22be14aaab2f4ac89baf1bf58bccc8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:08 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:15:08.833909 19376 ts_tablet_manager.cc:1367] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.010s
I20240502 14:15:08.834396 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet 22be14aaab2f4ac89baf1bf58bccc8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:08 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:08.835119 19376 raft_consensus.cc:348] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:08.835414 19376 raft_consensus.cc:374] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.835546 19376 raft_consensus.cc:729] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.835886 19376 consensus_queue.cc:260] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:08.838191 19376 ts_tablet_manager.cc:1398] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:08.838905 19155 raft_consensus.cc:482] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:08.839059 19155 raft_consensus.cc:504] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:08.839829 19155 leader_election.cc:288] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:08.840235 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7db78325b44450a87520a14a2444981" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:08.840749 16265 tablet_service.cc:1514] Processing DeleteTablet for tablet 22be14aaab2f4ac89baf1bf58bccc8f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:08 UTC) from {username='slave'} at 127.0.0.1:42272
W20240502 14:15:08.841225 13383 leader_election.cc:341] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:08.843276 19407 tablet_replica.cc:330] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:08.843570 19407 raft_consensus.cc:2229] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:08.843883 19407 raft_consensus.cc:2258] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:08.849598 19407 ts_tablet_manager.cc:1875] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:08.850279 19374 tablet_bootstrap.cc:492] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:08.850980 19406 tablet_replica.cc:330] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:08.851246 19406 raft_consensus.cc:2229] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:08.851615 19406 raft_consensus.cc:2258] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:08.852479 19374 tablet_bootstrap.cc:654] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.853865 19404 tablet_replica.cc:330] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:15:08.854118 19404 raft_consensus.cc:2229] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:08.854342 19404 raft_consensus.cc:2258] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:08.819062 13319 consensus_queue.cc:237] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:08.825567 19022 consensus_queue.cc:1035] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:08.840456 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7db78325b44450a87520a14a2444981" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:08.856007 16419 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:08.856122 16022 raft_consensus.cc:1261] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:08.856523 19397 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:08.856877 19024 consensus_queue.cc:1035] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:08.857023 19381 raft_consensus.cc:374] T d7db78325b44450a87520a14a2444981 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.857121 19381 raft_consensus.cc:729] T d7db78325b44450a87520a14a2444981 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.857328 19382 ts_tablet_manager.cc:1398] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.071s user 0.002s sys 0.000s
I20240502 14:15:08.858494 19407 ts_tablet_manager.cc:1888] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:08.859423 19406 ts_tablet_manager.cc:1875] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:08.857568 19381 consensus_queue.cc:260] T d7db78325b44450a87520a14a2444981 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:08.859947 13318 raft_consensus.cc:2454] T d7db78325b44450a87520a14a2444981 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:15:08.861259 13384 leader_election.cc:302] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:08.861548 19155 raft_consensus.cc:2781] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:08.861608 19155 raft_consensus.cc:482] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:08.861668 19155 raft_consensus.cc:3037] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.863552 19024 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.864192 19155 raft_consensus.cc:504] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:08.865097 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7db78325b44450a87520a14a2444981" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
W20240502 14:15:08.865523 13383 leader_election.cc:341] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:08.866855 19407 log.cc:1196] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/22be14aaab2f4ac89baf1bf58bccc8f1
I20240502 14:15:08.867237 19155 leader_election.cc:288] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:08.869315 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d5975e02205a43d5b93ed8ab51f8a0f8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:08.871160 16022 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.872396 19407 ts_tablet_manager.cc:1909] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:15:08.873330 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7db78325b44450a87520a14a2444981" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:08.873453 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 22be14aaab2f4ac89baf1bf58bccc8f1 (table rebalancer_dynamic_table_new_name [id=0db2138ef18044cf9d18f9c87597b57e]) successfully deleted
I20240502 14:15:08.873515 13318 raft_consensus.cc:3037] T d7db78325b44450a87520a14a2444981 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.876119 13318 raft_consensus.cc:2454] T d7db78325b44450a87520a14a2444981 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:08.877568 19380 tablet_bootstrap.cc:492] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:08.879631 19380 tablet_bootstrap.cc:654] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.882616 13447 raft_consensus.cc:1261] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:15:08.886030 13384 leader_election.cc:302] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:08.886502 19154 raft_consensus.cc:2781] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:08.887269 19022 consensus_queue.cc:1035] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:08.888655 13580 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.894917 13045 catalog_manager.cc:5484] T d5975e02205a43d5b93ed8ab51f8a0f8 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.896539 19391 raft_consensus.cc:2932] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:08.897583 16419 raft_consensus.cc:2932] T d5975e02205a43d5b93ed8ab51f8a0f8 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:08.900265 19374 tablet_bootstrap.cc:492] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:08.900460 19374 ts_tablet_manager.cc:1367] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.002s
I20240502 14:15:08.901340 19374 raft_consensus.cc:348] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:08.901546 19374 raft_consensus.cc:374] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.901700 19374 raft_consensus.cc:729] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.902277 19404 ts_tablet_manager.cc:1875] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:08.903087 18788 raft_consensus.cc:482] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:08.903235 18788 raft_consensus.cc:504] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:08.903923 18788 leader_election.cc:288] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:08.906515 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac1154570d4420ae8764ca18667f29" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:08.906894 16022 raft_consensus.cc:2454] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:08.907332 16221 leader_election.cc:302] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:08.907586 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac1154570d4420ae8764ca18667f29" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:08.907878 19176 raft_consensus.cc:2781] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:08.907984 19176 raft_consensus.cc:482] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:08.907953 13318 raft_consensus.cc:2454] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:08.908074 19176 raft_consensus.cc:3037] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.908309 19374 consensus_queue.cc:260] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:08.909816 19380 tablet_bootstrap.cc:492] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:08.909948 19380 ts_tablet_manager.cc:1367] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240502 14:15:08.910653 19176 raft_consensus.cc:504] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:08.912762 19176 leader_election.cc:288] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:08.913587 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac1154570d4420ae8764ca18667f29" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:08.913796 13318 raft_consensus.cc:3037] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.913821 16022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac1154570d4420ae8764ca18667f29" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:08.914017 16022 raft_consensus.cc:3037] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:08.915633 16022 raft_consensus.cc:2454] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:08.915998 16221 leader_election.cc:302] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:08.916317 19176 raft_consensus.cc:2781] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:08.916873 19176 raft_consensus.cc:686] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:08.917290 19176 consensus_queue.cc:237] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:08.917991 13447 raft_consensus.cc:2932] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:08.918577 19374 ts_tablet_manager.cc:1398] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.018s user 0.000s sys 0.001s
I20240502 14:15:08.918815 19374 tablet_bootstrap.cc:492] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:08.872808 16151 raft_consensus.cc:1261] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:15:08.920527 19380 raft_consensus.cc:348] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.925521 19380 raft_consensus.cc:374] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.925771 19380 raft_consensus.cc:729] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.923475 13560 tablet_service.cc:1514] Processing DeleteTablet for tablet d5975e02205a43d5b93ed8ab51f8a0f8 with delete_type TABLET_DATA_TOMBSTONED (TS 7eb4ecb2379b44c0954f2658c4a04350 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58526
I20240502 14:15:08.927748 19374 tablet_bootstrap.cc:654] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.920207 13318 raft_consensus.cc:2454] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:08.927336 19380 consensus_queue.cc:260] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.922422 19411 consensus_queue.cc:1035] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.006s
I20240502 14:15:08.924017 19381 ts_tablet_manager.cc:1398] T d7db78325b44450a87520a14a2444981 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.135s user 0.003s sys 0.000s
I20240502 14:15:08.943257 19380 ts_tablet_manager.cc:1398] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.033s user 0.003s sys 0.000s
I20240502 14:15:08.943603 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:08.944089 13318 consensus_queue.cc:237] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:08.947522 18849 raft_consensus.cc:1050] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER d1da7f11f0604f78934354a992bb0f55 to VOTER
I20240502 14:15:08.950302 18849 consensus_queue.cc:237] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:08.953649 16265 tablet_service.cc:1467] Processing CreateTablet for tablet 7212a46a0aa84f208351734c9ce3e17c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3ae4fd60ec804521ac7948f90b449ee5]), partition=
I20240502 14:15:08.954006 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7212a46a0aa84f208351734c9ce3e17c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.954483 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 7212a46a0aa84f208351734c9ce3e17c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3ae4fd60ec804521ac7948f90b449ee5]), partition=
I20240502 14:15:08.954746 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7212a46a0aa84f208351734c9ce3e17c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.957500 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 7212a46a0aa84f208351734c9ce3e17c (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3ae4fd60ec804521ac7948f90b449ee5]), partition=
I20240502 14:15:08.957784 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7212a46a0aa84f208351734c9ce3e17c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:08.960028 19022 raft_consensus.cc:1050] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 to VOTER
I20240502 14:15:08.961041 19022 consensus_queue.cc:237] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:08.971041 13445 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:08.971382 16151 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:08.971532 19382 tablet_bootstrap.cc:492] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:08.972191 18881 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:08.973376 19313 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:08.974383 19382 tablet_bootstrap.cc:654] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.979892 16285 raft_consensus.cc:1261] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:08.980659 19395 consensus_queue.cc:1035] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:08.984700 19382 tablet_bootstrap.cc:492] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:08.984937 19382 ts_tablet_manager.cc:1367] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240502 14:15:08.984958 19373 tablet_bootstrap.cc:492] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:08.986425 19382 raft_consensus.cc:348] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.986552 19404 ts_tablet_manager.cc:1888] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:08.986654 19404 log.cc:1196] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/22be14aaab2f4ac89baf1bf58bccc8f1
I20240502 14:15:08.986665 19382 raft_consensus.cc:374] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:08.986799 19373 tablet_bootstrap.cc:654] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:08.986809 19382 raft_consensus.cc:729] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:08.987077 19404 ts_tablet_manager.cc:1909] T 22be14aaab2f4ac89baf1bf58bccc8f1 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:15:08.988183 19382 consensus_queue.cc:260] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:08.989615 19382 ts_tablet_manager.cc:1398] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:08.991984 19154 raft_consensus.cc:686] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:08.992309 19154 consensus_queue.cc:237] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:08.994189 13445 raft_consensus.cc:1261] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:08.995764 19411 consensus_queue.cc:1035] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:09.000996 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 22be14aaab2f4ac89baf1bf58bccc8f1 (table rebalancer_dynamic_table_new_name [id=0db2138ef18044cf9d18f9c87597b57e]) successfully deleted
I20240502 14:15:09.003870 19404 tablet_replica.cc:330] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:15:09.004783 19404 raft_consensus.cc:2229] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:09.005651 16152 raft_consensus.cc:1261] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:15:09.006166 19406 ts_tablet_manager.cc:1888] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:09.005684 19404 raft_consensus.cc:2258] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:09.006317 19412 consensus_queue.cc:1035] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:09.008396 19406 log.cc:1196] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/22be14aaab2f4ac89baf1bf58bccc8f1
I20240502 14:15:09.008586 16022 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:09.008677 19395 raft_consensus.cc:2932] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:09.009177 19313 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:09.009315 19406 ts_tablet_manager.cc:1909] T 22be14aaab2f4ac89baf1bf58bccc8f1 P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:15:09.009112 16152 raft_consensus.cc:2932] T 3d39f694c8554638a3b10ebf7cd01350 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:09.010331 19404 ts_tablet_manager.cc:1875] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:09.015962 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 22be14aaab2f4ac89baf1bf58bccc8f1 (table rebalancer_dynamic_table_new_name [id=0db2138ef18044cf9d18f9c87597b57e]) successfully deleted
I20240502 14:15:09.029081 13188 raft_consensus.cc:2932] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:09.033205 19428 ts_tablet_manager.cc:902] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3: Initiating tablet copy from peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:09.037874 13043 catalog_manager.cc:5484] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:09.040386 19422 tablet_bootstrap.cc:492] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:09.043205 13186 raft_consensus.cc:1261] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:15:09.044564 16152 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:09.047564 16154 raft_consensus.cc:1261] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:09.048292 19411 consensus_queue.cc:1035] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:09.048549 18881 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:09.052233 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d39f694c8554638a3b10ebf7cd01350 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:09.054271 18788 raft_consensus.cc:482] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:09.054450 18788 raft_consensus.cc:504] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.055307 18788 leader_election.cc:288] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:09.055940 19422 tablet_bootstrap.cc:654] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.056367 16154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7212a46a0aa84f208351734c9ce3e17c" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
W20240502 14:15:09.056948 16220 leader_election.cc:341] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:09.057628 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7212a46a0aa84f208351734c9ce3e17c" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
W20240502 14:15:09.058364 16219 leader_election.cc:341] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:09.058490 16219 leader_election.cc:302] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5297e816f782430d9a01220f0471a882; no voters: 88b77f6edc1c4a72b548bf47c05704f3, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:09.058998 18788 raft_consensus.cc:2726] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:09.063680 19422 tablet_bootstrap.cc:492] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:09.063374 13042 catalog_manager.cc:5484] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 5 to 7, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 2 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:09.065788 19428 tablet_copy_client.cc:308] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.197:43695
I20240502 14:15:09.066110 19422 ts_tablet_manager.cc:1367] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20240502 14:15:09.066841 19422 raft_consensus.cc:348] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.067438 19422 raft_consensus.cc:374] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.067656 19422 raft_consensus.cc:729] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.070537 19374 tablet_bootstrap.cc:492] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:09.070780 19374 ts_tablet_manager.cc:1367] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.152s user 0.005s sys 0.009s
I20240502 14:15:09.070943 13042 catalog_manager.cc:5484] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:09.070997 16152 raft_consensus.cc:2932] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.071542 19391 raft_consensus.cc:2932] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.071748 19374 raft_consensus.cc:348] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.072110 19374 raft_consensus.cc:374] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.072216 19374 raft_consensus.cc:729] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.072474 19374 consensus_queue.cc:260] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.076750 19404 ts_tablet_manager.cc:1888] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:15:09.077396 19422 consensus_queue.cc:260] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.078126 19422 ts_tablet_manager.cc:1398] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240502 14:15:09.078660 19374 ts_tablet_manager.cc:1398] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 14:15:09.079085 16285 raft_consensus.cc:2932] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.076930 19404 log.cc:1196] T d5975e02205a43d5b93ed8ab51f8a0f8 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/d5975e02205a43d5b93ed8ab51f8a0f8
I20240502 14:15:09.080478 16020 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:09.082711 13046 catalog_manager.cc:5484] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:09.082801 13447 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:09.104328 13319 consensus_queue.cc:237] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:09.105499 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet d5975e02205a43d5b93ed8ab51f8a0f8 (table rebalance_test_table_13 [id=c0cb4710334148e4b976992ee52cd47c]) successfully deleted
I20240502 14:15:09.113677 16151 raft_consensus.cc:1261] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:09.114740 19397 consensus_queue.cc:1035] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:08.940254 16155 raft_consensus.cc:2932] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:09.119248 19412 consensus_queue.cc:1035] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:09.120309 13448 raft_consensus.cc:1261] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:09.120689 13446 raft_consensus.cc:2932] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.122697 19412 raft_consensus.cc:2932] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.126574 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1d607f6e65194399844bc14457fe8f15 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:09.130230 13046 catalog_manager.cc:5484] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:09.133795 13185 raft_consensus.cc:2932] T 3d39f694c8554638a3b10ebf7cd01350 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:09.134459 16265 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d607f6e65194399844bc14457fe8f15 with delete_type TABLET_DATA_TOMBSTONED (TS 5297e816f782430d9a01220f0471a882 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:09.134786 19407 tablet_replica.cc:330] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:09.135002 19407 raft_consensus.cc:2229] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:09.135263 19407 raft_consensus.cc:2258] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:09.136251 19407 ts_tablet_manager.cc:1875] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:09.142328 19397 consensus_queue.cc:1035] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:09.145093 19407 ts_tablet_manager.cc:1888] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:09.146142 19407 log.cc:1196] T 1d607f6e65194399844bc14457fe8f15 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/1d607f6e65194399844bc14457fe8f15
I20240502 14:15:09.147087 13446 raft_consensus.cc:2932] T 1d607f6e65194399844bc14457fe8f15 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.148011 16151 raft_consensus.cc:2932] T 1d607f6e65194399844bc14457fe8f15 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.148773 19284 raft_consensus.cc:482] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:09.148945 19284 raft_consensus.cc:504] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:09.149861 19284 leader_election.cc:288] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:09.150499 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7db78325b44450a87520a14a2444981" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:09.151181 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d7db78325b44450a87520a14a2444981" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:09.151521 13318 raft_consensus.cc:2379] T d7db78325b44450a87520a14a2444981 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8b47d4390f8343a493d6c00123785968 in current term 1: Already voted for candidate 88b77f6edc1c4a72b548bf47c05704f3 in this term.
I20240502 14:15:09.151979 16355 leader_election.cc:302] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b47d4390f8343a493d6c00123785968; no voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:09.152458 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 1d607f6e65194399844bc14457fe8f15 (table rebalancer_extra_table_10 [id=89586fac1cc04eb6b38593443f23b4c7]) successfully deleted
I20240502 14:15:09.152719 19284 raft_consensus.cc:3037] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.155282 19284 raft_consensus.cc:2726] T d7db78325b44450a87520a14a2444981 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240502 14:15:09.156319 13384 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.199:33857, user_credentials={real_user=slave}} blocked reactor thread for 140401us
I20240502 14:15:09.156548 19442 ts_tablet_manager.cc:902] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:09.157575 19373 tablet_bootstrap.cc:492] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:09.157693 19373 ts_tablet_manager.cc:1367] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.173s user 0.003s sys 0.000s
I20240502 14:15:09.158355 19373 raft_consensus.cc:348] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.158522 19373 raft_consensus.cc:374] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.158568 19373 raft_consensus.cc:729] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.158737 19373 consensus_queue.cc:260] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.159148 19446 raft_consensus.cc:482] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:09.159226 19373 ts_tablet_manager.cc:1398] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:09.159277 19446 raft_consensus.cc:504] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.160091 16043 tablet_copy_service.cc:140] P d1da7f11f0604f78934354a992bb0f55: Received BeginTabletCopySession request for tablet 8a98395ed74c4c2eb265a141aec14bd3 from peer 88b77f6edc1c4a72b548bf47c05704f3 ({username='slave'} at 127.6.196.195:47189)
I20240502 14:15:09.160270 16043 tablet_copy_service.cc:161] P d1da7f11f0604f78934354a992bb0f55: Beginning new tablet copy session on tablet 8a98395ed74c4c2eb265a141aec14bd3 from peer 88b77f6edc1c4a72b548bf47c05704f3 at {username='slave'} at 127.6.196.195:47189: session id = 88b77f6edc1c4a72b548bf47c05704f3-8a98395ed74c4c2eb265a141aec14bd3
I20240502 14:15:09.160786 19446 leader_election.cc:288] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:09.161505 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b46f71180f4269bee53f206e9c2665" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:09.161749 16419 raft_consensus.cc:2454] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:15:09.162217 13121 leader_election.cc:302] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:09.162536 19446 raft_consensus.cc:2781] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:09.162665 19446 raft_consensus.cc:482] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:09.162853 19446 raft_consensus.cc:3037] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.163115 16043 tablet_copy_source_session.cc:205] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:09.165544 19446 raft_consensus.cc:504] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.166653 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b46f71180f4269bee53f206e9c2665" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:09.166826 16419 raft_consensus.cc:3037] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.168008 19428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a98395ed74c4c2eb265a141aec14bd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.169567 16419 raft_consensus.cc:2454] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:15:09.170011 13121 leader_election.cc:302] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:09.170301 19278 raft_consensus.cc:2781] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:09.173415 19278 raft_consensus.cc:686] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:15:09.173696 19278 consensus_queue.cc:237] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.174816 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d39f694c8554638a3b10ebf7cd01350 with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:09.175010 19428 tablet_copy_client.cc:783] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:09.175176 19428 tablet_copy_client.cc:647] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:09.177659 13043 catalog_manager.cc:5484] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:09.178006 13042 catalog_manager.cc:5484] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:09.158670 13046 catalog_manager.cc:5484] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:09.179808 19428 tablet_copy_client.cc:523] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:09.180078 19446 leader_election.cc:288] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:09.182752 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b46f71180f4269bee53f206e9c2665" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:09.182948 16288 raft_consensus.cc:2454] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:15:09.183782 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76b46f71180f4269bee53f206e9c2665" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:09.183943 16288 raft_consensus.cc:3037] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.184672 19428 tablet_bootstrap.cc:492] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:09.186290 16288 raft_consensus.cc:2454] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:15:09.194463 13580 consensus_queue.cc:237] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:09.197405 16155 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:09.198190 19437 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:09.201885 19421 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:09.202576 16152 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:09.206135 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 652e49046212458280095c016a342f47 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:09.211647 13046 catalog_manager.cc:5484] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 8 to 9, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:09.215260 19428 tablet_bootstrap.cc:492] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:09.215610 19428 tablet_bootstrap.cc:492] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap complete.
I20240502 14:15:09.215809 19428 ts_tablet_manager.cc:1367] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.015s
I20240502 14:15:09.216435 19428 raft_consensus.cc:348] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:09.216626 19428 raft_consensus.cc:729] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: LEARNER
I20240502 14:15:09.216768 19428 consensus_queue.cc:260] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:09.217283 19428 ts_tablet_manager.cc:1398] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:15:09.218304 13429 tablet_service.cc:1514] Processing DeleteTablet for tablet 652e49046212458280095c016a342f47 with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:09.219631 16043 tablet_copy_service.cc:342] P d1da7f11f0604f78934354a992bb0f55: Request end of tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-8a98395ed74c4c2eb265a141aec14bd3 received from {username='slave'} at 127.6.196.195:47189
I20240502 14:15:09.219766 16043 tablet_copy_service.cc:433] P d1da7f11f0604f78934354a992bb0f55: ending tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-8a98395ed74c4c2eb265a141aec14bd3 on tablet 8a98395ed74c4c2eb265a141aec14bd3 with peer 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:15:09.222913 19442 tablet_copy_client.cc:308] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:15:09.223531 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet e17e489fe5e04a758d9d9d0259159c64 from peer 88b77f6edc1c4a72b548bf47c05704f3 ({username='slave'} at 127.6.196.195:49767)
I20240502 14:15:09.223729 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet e17e489fe5e04a758d9d9d0259159c64 from peer 88b77f6edc1c4a72b548bf47c05704f3 at {username='slave'} at 127.6.196.195:49767: session id = 88b77f6edc1c4a72b548bf47c05704f3-e17e489fe5e04a758d9d9d0259159c64
I20240502 14:15:09.224404 16023 raft_consensus.cc:1261] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:09.225059 19421 consensus_queue.cc:1035] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:09.227689 19452 tablet_replica.cc:330] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:09.227689 13601 tablet_copy_source_session.cc:205] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:09.227934 19452 raft_consensus.cc:2229] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:09.228150 19452 pending_rounds.cc:62] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: Trying to abort 1 pending ops.
I20240502 14:15:09.228291 19452 pending_rounds.cc:68] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: Aborting op as it isn't in flight: id { term: 2 index: 6 } timestamp: 7023244526763470848 op_type: NO_OP noop_request { }
I20240502 14:15:09.228500 19452 raft_consensus.cc:2866] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240502 14:15:09.228655 19452 raft_consensus.cc:2258] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:09.228948 16023 raft_consensus.cc:2932] T 652e49046212458280095c016a342f47 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:09.234840 19442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17e489fe5e04a758d9d9d0259159c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.242177 16288 raft_consensus.cc:1261] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 2 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:09.242715 19171 consensus_queue.cc:1035] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:09.255575 19452 ts_tablet_manager.cc:1875] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:09.259384 19154 raft_consensus.cc:482] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:09.259513 19154 raft_consensus.cc:504] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.260226 19154 leader_election.cc:288] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:09.261619 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7212a46a0aa84f208351734c9ce3e17c" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:09.261822 16288 raft_consensus.cc:2454] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:15:09.261847 16419 raft_consensus.cc:1261] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 2 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:09.262300 13384 leader_election.cc:302] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:09.262871 19154 raft_consensus.cc:2781] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:09.262995 19154 raft_consensus.cc:482] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:09.262893 18668 consensus_queue.cc:1035] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:09.263092 19154 raft_consensus.cc:3037] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.267624 19154 raft_consensus.cc:504] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.268623 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7212a46a0aa84f208351734c9ce3e17c" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:09.268815 16152 raft_consensus.cc:3037] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.271857 16152 raft_consensus.cc:2454] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:09.272394 13383 leader_election.cc:302] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:09.273751 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7212a46a0aa84f208351734c9ce3e17c" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:09.273949 16288 raft_consensus.cc:3037] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.276396 16288 raft_consensus.cc:2454] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:09.276630 19452 ts_tablet_manager.cc:1888] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:15:09.276829 19452 log.cc:1196] T 3d39f694c8554638a3b10ebf7cd01350 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/3d39f694c8554638a3b10ebf7cd01350
I20240502 14:15:09.276890 19154 leader_election.cc:288] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:09.277437 19154 raft_consensus.cc:2781] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:09.277776 19154 raft_consensus.cc:686] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:09.277858 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 3d39f694c8554638a3b10ebf7cd01350 (table rebalance_test_table_13 [id=c0cb4710334148e4b976992ee52cd47c]) successfully deleted
I20240502 14:15:09.277915 19452 tablet_replica.cc:330] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:09.278241 19452 raft_consensus.cc:2229] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:09.278206 19154 consensus_queue.cc:237] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.278491 19452 raft_consensus.cc:2258] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:09.285184 19452 ts_tablet_manager.cc:1875] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:09.286736 13042 catalog_manager.cc:5484] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:09.296898 16151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7212a46a0aa84f208351734c9ce3e17c" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:09.297461 16151 raft_consensus.cc:2362] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1. Re-sending same reply.
I20240502 14:15:09.310173 19452 ts_tablet_manager.cc:1888] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:15:09.313298 18788 consensus_queue.cc:1035] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.313654 19452 log.cc:1196] T 652e49046212458280095c016a342f47 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/652e49046212458280095c016a342f47
I20240502 14:15:09.314723 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 652e49046212458280095c016a342f47 (table rebalance_test_table_11 [id=b7310fe1c01b4fe7b29328321c07e72d]) successfully deleted
I20240502 14:15:09.332370 13185 raft_consensus.cc:1261] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:09.333098 19171 consensus_queue.cc:1035] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:09.348048 19442 tablet_copy_client.cc:783] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:09.348304 19442 tablet_copy_client.cc:647] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 1 WAL segments...
W20240502 14:15:09.347648 16355 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.194:33193, user_credentials={real_user=slave}} blocked reactor thread for 100534us
I20240502 14:15:09.350885 19442 tablet_copy_client.cc:523] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:09.356141 18788 consensus_queue.cc:1035] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.366155 19442 tablet_bootstrap.cc:492] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:09.407996 13447 raft_consensus.cc:1203] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:09.410441 19442 tablet_bootstrap.cc:492] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:09.410888 19442 tablet_bootstrap.cc:492] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap complete.
I20240502 14:15:09.411156 19442 ts_tablet_manager.cc:1367] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.000s
I20240502 14:15:09.411950 19442 raft_consensus.cc:348] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:09.412123 19442 raft_consensus.cc:729] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: LEARNER
I20240502 14:15:09.412339 19442 consensus_queue.cc:260] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:09.412735 19442 ts_tablet_manager.cc:1398] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:09.414873 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-e17e489fe5e04a758d9d9d0259159c64 received from {username='slave'} at 127.6.196.195:49767
I20240502 14:15:09.414994 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-e17e489fe5e04a758d9d9d0259159c64 on tablet e17e489fe5e04a758d9d9d0259159c64 with peer 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:15:09.423911 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:09.432921 13318 consensus_queue.cc:237] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
W20240502 14:15:09.443360 13253 consensus_peers.cc:485] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a95b9049a344b98be91b9961f3e5bee. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:09.444123 16288 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:09.444232 13447 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:09.444705 19397 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:09.446566 16151 raft_consensus.cc:1261] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:09.446976 19397 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:09.447520 19444 consensus_queue.cc:1035] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.451231 16288 raft_consensus.cc:1261] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:09.451746 19443 consensus_queue.cc:1035] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.477500 13444 raft_consensus.cc:1203] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:09.486984 13043 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 7212a46a0aa84f208351734c9ce3e17c (table rebalancer_dynamic_table [id=3ae4fd60ec804521ac7948f90b449ee5]). Expected version 1 got 0
I20240502 14:15:09.487821 19484 tablet.cc:1722] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:09.488348 19484 tablet.cc:1620] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:09.494334 13043 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=3ae4fd60ec804521ac7948f90b449ee5] alter complete (version 1)
I20240502 14:15:09.494347 19483 tablet.cc:1722] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:09.494917 19483 tablet.cc:1620] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: MemRowSet was empty: no flush needed.
I20240502 14:15:09.494839 19397 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:09.500097 13043 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet 7212a46a0aa84f208351734c9ce3e17c (table rebalancer_dynamic_table [id=3ae4fd60ec804521ac7948f90b449ee5]). Expected version 1 got 0
I20240502 14:15:09.502627 13447 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:09.508723 16287 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:09.508922 13044 catalog_manager.cc:5484] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:09.513653 19485 tablet.cc:1722] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:09.514452 19485 tablet.cc:1620] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:09.520426 13046 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet 7212a46a0aa84f208351734c9ce3e17c (table rebalancer_dynamic_table [id=3ae4fd60ec804521ac7948f90b449ee5]). Expected version 1 got 0
I20240502 14:15:09.537313 19444 consensus_queue.cc:1035] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.542563 19155 consensus_queue.cc:1035] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.580511 19446 consensus_queue.cc:1035] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.586441 19450 consensus_queue.cc:1035] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:09.614202 13046 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:09.624816 19483 tablet.cc:1722] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:09.625278 19483 tablet.cc:1620] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: MemRowSet was empty: no flush needed.
I20240502 14:15:09.632429 19485 tablet.cc:1722] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:09.632815 19485 tablet.cc:1620] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:09.638316 19484 tablet.cc:1722] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:09.638729 19484 tablet.cc:1620] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:09.644791 13043 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=3ae4fd60ec804521ac7948f90b449ee5] alter complete (version 2)
I20240502 14:15:09.662845 13318 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6395ce4c30eb4644b0a711de3e60b9c0"
dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:42424
I20240502 14:15:09.663025 13318 raft_consensus.cc:595] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:09.689296 16418 consensus_queue.cc:237] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:09.691484 16288 raft_consensus.cc:1261] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:09.691586 16353 consensus_peers.cc:485] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fc251d4e82e452d989405e121d08dd4. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:09.691608 16151 raft_consensus.cc:1261] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:09.691993 19460 consensus_queue.cc:1035] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:09.692267 19461 consensus_queue.cc:1035] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:09.694747 19461 raft_consensus.cc:2932] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:09.695214 16151 raft_consensus.cc:2932] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:09.695309 16288 raft_consensus.cc:2932] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:09.699653 13044 catalog_manager.cc:5484] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:09.716224 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:09.717118 13447 consensus_queue.cc:237] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:09.719220 16418 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:15:09.719816 13384 consensus_peers.cc:485] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c3e9160c46d64fa0a964cc39aec91dfd. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:09.719861 13318 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:09.720029 19155 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:09.721491 19155 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:09.724221 19155 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:09.727666 13318 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:09.729158 16265 tablet_service.cc:1467] Processing CreateTablet for tablet eb472259eca84c31b2ff9f33f7e61177 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=8f0281d7a3a049329cb3130e1f8d0c04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.729552 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb472259eca84c31b2ff9f33f7e61177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.729552 16401 tablet_service.cc:1467] Processing CreateTablet for tablet eb472259eca84c31b2ff9f33f7e61177 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=8f0281d7a3a049329cb3130e1f8d0c04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.729492 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 2363a23cdde54893ae807ef3960dd032 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=8f0281d7a3a049329cb3130e1f8d0c04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.729902 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb472259eca84c31b2ff9f33f7e61177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.730080 16400 tablet_service.cc:1467] Processing CreateTablet for tablet c1b7405d7ff34356b0aa6ecb12b50817 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=8f0281d7a3a049329cb3130e1f8d0c04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.730293 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2363a23cdde54893ae807ef3960dd032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.730484 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1b7405d7ff34356b0aa6ecb12b50817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.733924 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 2363a23cdde54893ae807ef3960dd032 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=8f0281d7a3a049329cb3130e1f8d0c04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.734289 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2363a23cdde54893ae807ef3960dd032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.735611 15999 tablet_service.cc:1467] Processing CreateTablet for tablet eb472259eca84c31b2ff9f33f7e61177 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=8f0281d7a3a049329cb3130e1f8d0c04]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.735994 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb472259eca84c31b2ff9f33f7e61177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.737612 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 2363a23cdde54893ae807ef3960dd032 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=8f0281d7a3a049329cb3130e1f8d0c04]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.738045 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2363a23cdde54893ae807ef3960dd032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.744153 13168 tablet_service.cc:1467] Processing CreateTablet for tablet c1b7405d7ff34356b0aa6ecb12b50817 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=8f0281d7a3a049329cb3130e1f8d0c04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.744452 13560 tablet_service.cc:1467] Processing CreateTablet for tablet c1b7405d7ff34356b0aa6ecb12b50817 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=8f0281d7a3a049329cb3130e1f8d0c04]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:09.744568 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1b7405d7ff34356b0aa6ecb12b50817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.744809 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1b7405d7ff34356b0aa6ecb12b50817. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:09.749012 13043 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:09.750733 19558 tablet_bootstrap.cc:492] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:09.758630 19558 tablet_bootstrap.cc:654] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.759354 13046 catalog_manager.cc:5484] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:09.762392 19483 tablet.cc:1722] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:09.763020 19484 tablet.cc:1722] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:09.774448 19485 tablet.cc:1722] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:09.778038 19558 tablet_bootstrap.cc:492] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:09.778257 19558 ts_tablet_manager.cc:1367] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.005s
I20240502 14:15:09.779237 19558 raft_consensus.cc:348] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:09.779610 19558 raft_consensus.cc:374] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.779783 19558 raft_consensus.cc:729] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.780275 19558 consensus_queue.cc:260] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:09.782135 16023 consensus_queue.cc:237] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:09.783037 19558 ts_tablet_manager.cc:1398] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:15:09.784550 13185 raft_consensus.cc:1261] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:09.785061 16420 raft_consensus.cc:1261] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 2 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:09.785135 19171 consensus_queue.cc:1035] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:09.786679 19171 consensus_queue.cc:1035] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:09.787600 13046 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=3ae4fd60ec804521ac7948f90b449ee5] alter complete (version 3)
I20240502 14:15:09.788654 19171 raft_consensus.cc:2932] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:09.792920 16420 raft_consensus.cc:2932] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:09.794433 13185 raft_consensus.cc:2932] T 7da975869e82479b91fdda30680e6b3c P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:09.800495 13044 catalog_manager.cc:5484] T 7da975869e82479b91fdda30680e6b3c P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 3 to 5, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New cstate: current_term: 2 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:09.800906 19559 tablet_bootstrap.cc:492] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:09.802978 19559 tablet_bootstrap.cc:654] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.803642 19173 raft_consensus.cc:1050] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55: attempting to promote NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 to VOTER
I20240502 14:15:09.804437 19173 consensus_queue.cc:237] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:09.805066 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet 7da975869e82479b91fdda30680e6b3c with delete_type TABLET_DATA_TOMBSTONED (TS 5297e816f782430d9a01220f0471a882 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:09.814764 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7da975869e82479b91fdda30680e6b3c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:09.820473 19557 tablet_bootstrap.cc:492] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:09.824697 19557 tablet_bootstrap.cc:654] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.834456 19557 tablet_bootstrap.cc:492] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:09.835115 19557 ts_tablet_manager.cc:1367] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.006s
I20240502 14:15:09.835208 13447 raft_consensus.cc:1261] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:09.836143 18668 consensus_queue.cc:1035] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:09.837306 16151 raft_consensus.cc:1261] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:09.837368 16420 raft_consensus.cc:1261] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:09.838266 18668 consensus_queue.cc:1035] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:09.838590 19566 consensus_queue.cc:1035] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:09.842065 19565 tablet_bootstrap.cc:492] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:09.849622 19565 tablet_bootstrap.cc:654] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.849834 19559 tablet_bootstrap.cc:492] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:09.852288 19559 ts_tablet_manager.cc:1367] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.051s user 0.010s sys 0.000s
I20240502 14:15:09.852543 19556 tablet_bootstrap.cc:492] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:09.854913 19559 raft_consensus.cc:348] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:09.855336 19559 raft_consensus.cc:374] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.855441 19559 raft_consensus.cc:729] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.855510 19562 tablet_replica.cc:330] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:09.855664 19559 consensus_queue.cc:260] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:09.856784 19562 raft_consensus.cc:2229] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:09.860224 16418 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:09.861443 19562 raft_consensus.cc:2258] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:09.862489 19562 ts_tablet_manager.cc:1875] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:09.869607 19559 ts_tablet_manager.cc:1398] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240502 14:15:09.872179 19397 raft_consensus.cc:979] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d: : Instructing follower 88b77f6edc1c4a72b548bf47c05704f3 to start an election
I20240502 14:15:09.872324 19397 raft_consensus.cc:1067] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Signalling peer 88b77f6edc1c4a72b548bf47c05704f3 to start an election
I20240502 14:15:09.873236 13444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6395ce4c30eb4644b0a711de3e60b9c0"
dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
from {username='slave'} at 127.6.196.194:48133
I20240502 14:15:09.873505 19565 tablet_bootstrap.cc:492] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:09.873488 13444 raft_consensus.cc:482] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:09.873628 13444 raft_consensus.cc:3037] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:09.873735 19565 ts_tablet_manager.cc:1367] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240502 14:15:09.874722 19171 raft_consensus.cc:2932] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.877278 19562 ts_tablet_manager.cc:1888] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 14:15:09.877393 19562 log.cc:1196] T 7da975869e82479b91fdda30680e6b3c P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/7da975869e82479b91fdda30680e6b3c
I20240502 14:15:09.878480 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 7da975869e82479b91fdda30680e6b3c (table rebalancer_extra_table_4 [id=cb136ab1e407401daef8e9856e8ee6f9]) successfully deleted
I20240502 14:15:09.878942 13447 raft_consensus.cc:2932] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.879755 13444 raft_consensus.cc:504] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:09.880831 13444 leader_election.cc:288] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:09.881096 19557 raft_consensus.cc:348] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.881371 19557 raft_consensus.cc:374] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.881459 19557 raft_consensus.cc:729] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.881633 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6395ce4c30eb4644b0a711de3e60b9c0" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:09.881676 19557 consensus_queue.cc:260] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.881826 16152 raft_consensus.cc:3037] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:09.884193 16152 raft_consensus.cc:2454] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 2.
I20240502 14:15:09.887445 16152 raft_consensus.cc:2932] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.888314 16420 raft_consensus.cc:2932] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.895176 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6395ce4c30eb4644b0a711de3e60b9c0" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:09.895011 13042 catalog_manager.cc:5484] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.895355 13185 raft_consensus.cc:3037] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:09.897337 16152 raft_consensus.cc:1226] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Rejecting Update request from peer cd67493f6f0142f896010f2b8a39887d for earlier term 1. Current term is 2. Ops: []
I20240502 14:15:09.898062 13185 raft_consensus.cc:2454] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 2.
I20240502 14:15:09.899595 19395 consensus_queue.cc:1046] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:09.900187 13046 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:09.900346 19395 raft_consensus.cc:3032] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:09.900411 13046 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:09.900441 19395 raft_consensus.cc:729] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:09.900862 19395 consensus_queue.cc:260] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:09.901364 19395 raft_consensus.cc:3037] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:09.902130 19557 ts_tablet_manager.cc:1398] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.067s user 0.002s sys 0.000s
I20240502 14:15:09.902465 19557 tablet_bootstrap.cc:492] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:09.903941 19557 tablet_bootstrap.cc:654] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.904603 16020 consensus_queue.cc:237] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:09.907835 13046 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 7212a46a0aa84f208351734c9ce3e17c
I20240502 14:15:09.908474 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6395ce4c30eb4644b0a711de3e60b9c0" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:09.908946 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet 7212a46a0aa84f208351734c9ce3e17c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:09 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:09.909044 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 7212a46a0aa84f208351734c9ce3e17c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:09 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:09.909464 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet 7212a46a0aa84f208351734c9ce3e17c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:09 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:09.909699 19562 tablet_replica.cc:330] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:09.909982 19562 raft_consensus.cc:2229] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:09.910233 19562 raft_consensus.cc:2258] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:09.911185 19562 ts_tablet_manager.cc:1875] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:09.912109 13318 raft_consensus.cc:2454] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 2.
I20240502 14:15:09.913556 13384 leader_election.cc:302] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:09.913969 19443 raft_consensus.cc:2781] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:09.914112 19443 raft_consensus.cc:686] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:09.914350 19443 consensus_queue.cc:237] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:09.916533 19562 ts_tablet_manager.cc:1888] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:09.916647 19562 log.cc:1196] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/7212a46a0aa84f208351734c9ce3e17c
I20240502 14:15:09.917026 19562 ts_tablet_manager.cc:1909] T 7212a46a0aa84f208351734c9ce3e17c P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:15:09.918310 13447 raft_consensus.cc:1261] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:09.918845 19565 raft_consensus.cc:348] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:09.919124 19565 raft_consensus.cc:374] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.919495 19564 consensus_queue.cc:1035] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:09.919868 19565 raft_consensus.cc:729] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.920444 19565 consensus_queue.cc:260] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:09.921994 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 7212a46a0aa84f208351734c9ce3e17c (table rebalancer_dynamic_table_new_name [id=3ae4fd60ec804521ac7948f90b449ee5]) successfully deleted
I20240502 14:15:09.923048 19565 ts_tablet_manager.cc:1398] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.049s user 0.002s sys 0.000s
I20240502 14:15:09.923391 19565 tablet_bootstrap.cc:492] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:09.857900 19556 tablet_bootstrap.cc:654] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.931088 19461 raft_consensus.cc:482] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:09.931284 19461 raft_consensus.cc:504] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.930055 13043 catalog_manager.cc:5484] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 1 to 2, leader changed from cd67493f6f0142f896010f2b8a39887d (127.6.196.194) to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 2 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:09.931869 19461 leader_election.cc:288] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:09.932494 16152 raft_consensus.cc:1261] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:09.933180 13447 raft_consensus.cc:2932] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.933749 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2363a23cdde54893ae807ef3960dd032" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
W20240502 14:15:09.934767 16355 leader_election.cc:341] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:09.930300 19564 raft_consensus.cc:2932] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.933094 19173 consensus_queue.cc:1035] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:09.937649 16151 raft_consensus.cc:2932] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.939394 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8a98395ed74c4c2eb265a141aec14bd3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:09.935150 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2363a23cdde54893ae807ef3960dd032" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
W20240502 14:15:09.943578 16355 leader_election.cc:341] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:09.943702 16355 leader_election.cc:302] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b47d4390f8343a493d6c00123785968; no voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:09.944028 19461 raft_consensus.cc:2726] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:09.946107 19565 tablet_bootstrap.cc:654] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.946839 19576 tablet_replica.cc:330] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:09.947078 19576 raft_consensus.cc:2229] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:09.947414 19576 raft_consensus.cc:2258] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:09.948333 19576 ts_tablet_manager.cc:1875] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:09.957820 13042 catalog_manager.cc:5484] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:09.961941 16399 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a98395ed74c4c2eb265a141aec14bd3 with delete_type TABLET_DATA_TOMBSTONED (TS 8b47d4390f8343a493d6c00123785968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:09.964841 19557 tablet_bootstrap.cc:492] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:09.965024 19557 ts_tablet_manager.cc:1367] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.013s
I20240502 14:15:09.965924 19557 raft_consensus.cc:348] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:09.966157 19557 raft_consensus.cc:374] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.966224 19557 raft_consensus.cc:729] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.966464 19557 consensus_queue.cc:260] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:09.967079 19557 ts_tablet_manager.cc:1398] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:09.968449 19557 tablet_bootstrap.cc:492] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:09.968700 19575 tablet_replica.cc:330] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:09.970041 19557 tablet_bootstrap.cc:654] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.971653 19461 raft_consensus.cc:482] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:09.971808 19461 raft_consensus.cc:504] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:09.972581 19461 leader_election.cc:288] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:09.973294 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1b7405d7ff34356b0aa6ecb12b50817" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:09.973510 19556 tablet_bootstrap.cc:492] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:09.973527 13578 raft_consensus.cc:2454] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:15:09.973627 19556 ts_tablet_manager.cc:1367] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.121s user 0.003s sys 0.000s
I20240502 14:15:09.974296 16353 leader_election.cc:302] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:09.974275 19556 raft_consensus.cc:348] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:09.974514 19556 raft_consensus.cc:374] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:09.974625 19556 raft_consensus.cc:729] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:09.974721 19461 raft_consensus.cc:2781] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:09.974838 19461 raft_consensus.cc:482] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:09.974928 19461 raft_consensus.cc:3037] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.974896 19556 consensus_queue.cc:260] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:09.975704 19575 raft_consensus.cc:2229] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:09.977263 19575 raft_consensus.cc:2258] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:09.978008 19556 ts_tablet_manager.cc:1398] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:15:09.978288 19556 tablet_bootstrap.cc:492] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:09.979993 19575 ts_tablet_manager.cc:1875] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:09.980975 19556 tablet_bootstrap.cc:654] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:09.986858 19576 ts_tablet_manager.cc:1888] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:09.986987 19576 log.cc:1196] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/7212a46a0aa84f208351734c9ce3e17c
I20240502 14:15:09.986491 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1b7405d7ff34356b0aa6ecb12b50817" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:09.987466 19576 ts_tablet_manager.cc:1909] T 7212a46a0aa84f208351734c9ce3e17c P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:15:09.987533 13185 raft_consensus.cc:2454] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:15:09.989216 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 7212a46a0aa84f208351734c9ce3e17c (table rebalancer_dynamic_table_new_name [id=3ae4fd60ec804521ac7948f90b449ee5]) successfully deleted
I20240502 14:15:09.996503 19461 raft_consensus.cc:504] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:09.996918 19565 tablet_bootstrap.cc:492] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:09.997128 19565 ts_tablet_manager.cc:1367] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.074s user 0.007s sys 0.000s
I20240502 14:15:09.999481 19565 raft_consensus.cc:348] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:09.999694 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1b7405d7ff34356b0aa6ecb12b50817" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:09.999897 13185 raft_consensus.cc:3037] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:09.999960 19565 raft_consensus.cc:374] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.001874 19565 raft_consensus.cc:729] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.002415 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1b7405d7ff34356b0aa6ecb12b50817" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:10.002518 19461 leader_election.cc:288] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:10.002588 13578 raft_consensus.cc:3037] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.003221 19565 consensus_queue.cc:260] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:10.003942 19565 ts_tablet_manager.cc:1398] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:10.004676 13185 raft_consensus.cc:2454] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:15:10.005100 16353 leader_election.cc:302] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:10.005095 13578 raft_consensus.cc:2454] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:15:10.005434 19461 raft_consensus.cc:2781] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:10.005944 19461 raft_consensus.cc:686] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:15:10.006270 19461 consensus_queue.cc:237] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:10.010864 13044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:10.013953 19575 ts_tablet_manager.cc:1888] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:10.014891 19556 tablet_bootstrap.cc:492] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:10.015121 19556 ts_tablet_manager.cc:1367] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.000s
I20240502 14:15:10.016049 19556 raft_consensus.cc:348] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:10.017535 19556 raft_consensus.cc:374] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.017658 19556 raft_consensus.cc:729] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.017920 19556 consensus_queue.cc:260] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:10.022373 19556 ts_tablet_manager.cc:1398] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:15:10.023181 19575 log.cc:1196] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/7212a46a0aa84f208351734c9ce3e17c
I20240502 14:15:10.023733 19575 ts_tablet_manager.cc:1909] T 7212a46a0aa84f208351734c9ce3e17c P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:15:10.024628 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 7212a46a0aa84f208351734c9ce3e17c (table rebalancer_dynamic_table_new_name [id=3ae4fd60ec804521ac7948f90b449ee5]) successfully deleted
I20240502 14:15:10.027822 19557 tablet_bootstrap.cc:492] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:10.028015 19557 ts_tablet_manager.cc:1367] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.007s
I20240502 14:15:10.028705 19557 raft_consensus.cc:348] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:10.029371 19557 raft_consensus.cc:374] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.029505 19557 raft_consensus.cc:729] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.029767 19557 consensus_queue.cc:260] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:10.029210 19583 tablet_replica.cc:330] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:10.032271 19583 raft_consensus.cc:2229] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:10.032338 19557 ts_tablet_manager.cc:1398] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:15:10.032933 19583 raft_consensus.cc:2258] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:10.034636 19583 ts_tablet_manager.cc:1875] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:10.040853 13044 catalog_manager.cc:5484] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:10.041918 13298 tablet_service.cc:1467] Processing CreateTablet for tablet aa973b2d5ead4d7f89a5629c296a2bfd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ccb1096c99524de0b353beec634a3b56]), partition=
I20240502 14:15:10.042237 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa973b2d5ead4d7f89a5629c296a2bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.042255 13168 tablet_service.cc:1467] Processing CreateTablet for tablet aa973b2d5ead4d7f89a5629c296a2bfd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ccb1096c99524de0b353beec634a3b56]), partition=
I20240502 14:15:10.042572 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa973b2d5ead4d7f89a5629c296a2bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.045562 16135 tablet_service.cc:1467] Processing CreateTablet for tablet aa973b2d5ead4d7f89a5629c296a2bfd (DEFAULT_TABLE table=rebalancer_dynamic_table [id=ccb1096c99524de0b353beec634a3b56]), partition=
I20240502 14:15:10.045910 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa973b2d5ead4d7f89a5629c296a2bfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.050041 19437 raft_consensus.cc:1050] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 to VOTER
I20240502 14:15:10.050951 19437 consensus_queue.cc:237] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:10.054515 19583 ts_tablet_manager.cc:1888] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:10.054714 19583 log.cc:1196] T 8a98395ed74c4c2eb265a141aec14bd3 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/8a98395ed74c4c2eb265a141aec14bd3
I20240502 14:15:10.060833 19598 tablet_bootstrap.cc:492] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:10.063355 19598 tablet_bootstrap.cc:654] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.064695 19558 tablet_bootstrap.cc:492] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:10.066457 19558 tablet_bootstrap.cc:654] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.071484 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 8a98395ed74c4c2eb265a141aec14bd3 (table rebalancer_extra_table_12 [id=ef46a85b1ca942a9ae1f3a3aad6af281]) successfully deleted
I20240502 14:15:10.075408 13447 raft_consensus.cc:1261] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:10.075966 18849 consensus_queue.cc:1035] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:10.082208 19599 ts_tablet_manager.cc:902] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:10.084195 16020 raft_consensus.cc:1261] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:10.084703 18849 consensus_queue.cc:1035] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:10.085829 19558 tablet_bootstrap.cc:492] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:10.086061 19558 ts_tablet_manager.cc:1367] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.004s
I20240502 14:15:10.086530 13185 raft_consensus.cc:1261] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:10.086985 19558 raft_consensus.cc:348] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:10.087055 19437 consensus_queue.cc:1035] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:10.087319 19558 raft_consensus.cc:374] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.087466 19558 raft_consensus.cc:729] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.087790 19558 consensus_queue.cc:260] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:10.092571 19558 ts_tablet_manager.cc:1398] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 14:15:10.099844 19596 tablet_bootstrap.cc:492] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:10.101133 19596 tablet_bootstrap.cc:654] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.104718 19596 tablet_bootstrap.cc:492] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:10.104863 19596 ts_tablet_manager.cc:1367] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:10.105496 19596 raft_consensus.cc:348] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:10.107210 19596 raft_consensus.cc:374] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.107302 19596 raft_consensus.cc:729] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.107785 19596 consensus_queue.cc:260] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:10.109786 18849 raft_consensus.cc:2932] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:10.110419 13447 raft_consensus.cc:2932] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:10.118585 13044 catalog_manager.cc:5484] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:10.127089 16023 raft_consensus.cc:2932] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:10.145995 13185 raft_consensus.cc:2932] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:10.147665 19596 ts_tablet_manager.cc:1398] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.043s user 0.003s sys 0.000s
I20240502 14:15:10.154814 13578 consensus_queue.cc:237] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:10.159783 13447 raft_consensus.cc:1261] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:10.161059 19437 consensus_queue.cc:1035] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:10.161484 16023 raft_consensus.cc:1261] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:10.163648 19437 consensus_queue.cc:1035] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:10.164589 19421 raft_consensus.cc:2932] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:10.165094 13447 raft_consensus.cc:2932] T e17e489fe5e04a758d9d9d0259159c64 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:10.166934 16023 raft_consensus.cc:2932] T e17e489fe5e04a758d9d9d0259159c64 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:10.169047 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e17e489fe5e04a758d9d9d0259159c64 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:10.172734 19564 raft_consensus.cc:482] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:10.172895 19564 raft_consensus.cc:504] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:10.173903 19564 leader_election.cc:288] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:10.174679 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb472259eca84c31b2ff9f33f7e61177" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:10.174944 16420 raft_consensus.cc:2454] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:10.175585 15956 leader_election.cc:302] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:10.176831 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb472259eca84c31b2ff9f33f7e61177" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:10.177112 16287 raft_consensus.cc:2454] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:10.177945 19564 raft_consensus.cc:2781] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:10.178097 19564 raft_consensus.cc:482] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:10.178190 19564 raft_consensus.cc:3037] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.178277 13042 catalog_manager.cc:5484] T e17e489fe5e04a758d9d9d0259159c64 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:10.184022 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet e17e489fe5e04a758d9d9d0259159c64 with delete_type TABLET_DATA_TOMBSTONED (TS 20a967bad5504b1e9c6fb720ecdd1625 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37690
I20240502 14:15:10.189122 19564 raft_consensus.cc:504] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:10.190342 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb472259eca84c31b2ff9f33f7e61177" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:10.190624 16420 raft_consensus.cc:3037] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.197944 16420 raft_consensus.cc:2454] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:10.198392 15956 leader_election.cc:302] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:10.198655 19171 raft_consensus.cc:2781] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:10.198815 19171 raft_consensus.cc:686] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:10.199045 19171 consensus_queue.cc:237] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:10.201460 19598 tablet_bootstrap.cc:492] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:10.201550 19614 tablet_replica.cc:330] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:15:10.201628 19598 ts_tablet_manager.cc:1367] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.141s user 0.013s sys 0.056s
I20240502 14:15:10.201797 19614 raft_consensus.cc:2229] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:10.201980 19614 raft_consensus.cc:2258] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:10.202407 19598 raft_consensus.cc:348] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:10.202630 19598 raft_consensus.cc:374] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.202735 19598 raft_consensus.cc:729] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.202952 19598 consensus_queue.cc:260] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:10.203475 19598 ts_tablet_manager.cc:1398] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:10.203516 19614 ts_tablet_manager.cc:1875] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:10.209000 19564 leader_election.cc:288] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:10.209190 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb472259eca84c31b2ff9f33f7e61177" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:10.209437 16287 raft_consensus.cc:3037] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.212450 16287 raft_consensus.cc:2454] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:10.217604 19614 ts_tablet_manager.cc:1888] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:10.217743 19614 log.cc:1196] T e17e489fe5e04a758d9d9d0259159c64 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/e17e489fe5e04a758d9d9d0259159c64
I20240502 14:15:10.218405 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet e17e489fe5e04a758d9d9d0259159c64 (table rebalancer_extra_table_8 [id=296e981cee4f48689b6fcf85f9ab9562]) successfully deleted
I20240502 14:15:10.219053 19615 ts_tablet_manager.cc:902] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3: Initiating tablet copy from peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:10.220579 13044 catalog_manager.cc:5484] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:10.221369 19617 ts_tablet_manager.cc:902] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:10.231539 19171 raft_consensus.cc:482] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:10.231734 19599 tablet_copy_client.cc:308] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:15:10.231798 19171 raft_consensus.cc:504] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:10.232307 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 9a95b9049a344b98be91b9961f3e5bee from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:40779)
I20240502 14:15:10.232445 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 9a95b9049a344b98be91b9961f3e5bee from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:40779: session id = d1da7f11f0604f78934354a992bb0f55-9a95b9049a344b98be91b9961f3e5bee
I20240502 14:15:10.233048 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2363a23cdde54893ae807ef3960dd032" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:10.233319 16420 raft_consensus.cc:2454] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:10.233783 15956 leader_election.cc:302] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:10.234185 19564 raft_consensus.cc:2781] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:10.234284 19564 raft_consensus.cc:482] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:10.234397 19564 raft_consensus.cc:3037] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.237174 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2363a23cdde54893ae807ef3960dd032" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:10.237507 16287 raft_consensus.cc:2454] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:10.238296 13339 tablet_copy_source_session.cc:205] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:10.238314 19171 leader_election.cc:288] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:10.240154 19564 raft_consensus.cc:504] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:10.240437 19599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a95b9049a344b98be91b9961f3e5bee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.251782 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2363a23cdde54893ae807ef3960dd032" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:10.252043 16418 raft_consensus.cc:3037] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.252151 19599 tablet_copy_client.cc:783] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:10.252352 19599 tablet_copy_client.cc:647] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:10.255669 19599 tablet_copy_client.cc:523] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:10.255939 16418 raft_consensus.cc:2454] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:10.256716 15956 leader_election.cc:302] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:10.257103 19171 raft_consensus.cc:2781] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:10.257292 19171 raft_consensus.cc:686] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:10.257635 19171 consensus_queue.cc:237] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:10.259176 19599 tablet_bootstrap.cc:492] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:10.261399 19617 tablet_copy_client.cc:308] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:15:10.262219 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet c3e9160c46d64fa0a964cc39aec91dfd from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:53727)
I20240502 14:15:10.262357 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet c3e9160c46d64fa0a964cc39aec91dfd from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:53727: session id = d1da7f11f0604f78934354a992bb0f55-c3e9160c46d64fa0a964cc39aec91dfd
I20240502 14:15:10.264848 13470 tablet_copy_source_session.cc:205] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:10.266423 19617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e9160c46d64fa0a964cc39aec91dfd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.266204 13044 catalog_manager.cc:5484] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:10.272670 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2363a23cdde54893ae807ef3960dd032" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:10.272871 16287 raft_consensus.cc:3037] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.275681 16287 raft_consensus.cc:2454] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:10.276295 19564 leader_election.cc:288] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:10.289328 19615 tablet_copy_client.cc:308] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.200:43857
I20240502 14:15:10.289875 16442 tablet_copy_service.cc:140] P 8b47d4390f8343a493d6c00123785968: Received BeginTabletCopySession request for tablet 2fc251d4e82e452d989405e121d08dd4 from peer 88b77f6edc1c4a72b548bf47c05704f3 ({username='slave'} at 127.6.196.195:39073)
I20240502 14:15:10.290040 16442 tablet_copy_service.cc:161] P 8b47d4390f8343a493d6c00123785968: Beginning new tablet copy session on tablet 2fc251d4e82e452d989405e121d08dd4 from peer 88b77f6edc1c4a72b548bf47c05704f3 at {username='slave'} at 127.6.196.195:39073: session id = 88b77f6edc1c4a72b548bf47c05704f3-2fc251d4e82e452d989405e121d08dd4
I20240502 14:15:10.292686 16442 tablet_copy_source_session.cc:205] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:10.293718 19617 tablet_copy_client.cc:783] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:10.293845 19615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc251d4e82e452d989405e121d08dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.294070 19617 tablet_copy_client.cc:647] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:10.295075 13318 raft_consensus.cc:1261] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:15:10.295954 19443 consensus_queue.cc:1035] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:10.296044 19617 tablet_copy_client.cc:523] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:10.299424 19617 tablet_bootstrap.cc:492] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:10.300043 19615 tablet_copy_client.cc:783] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:10.300248 19615 tablet_copy_client.cc:647] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:10.307554 19615 tablet_copy_client.cc:523] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:10.312283 19615 tablet_bootstrap.cc:492] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:10.312891 19391 raft_consensus.cc:482] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:10.313032 19391 raft_consensus.cc:504] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:10.313763 19391 leader_election.cc:288] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:10.314333 16151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa973b2d5ead4d7f89a5629c296a2bfd" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:10.314520 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa973b2d5ead4d7f89a5629c296a2bfd" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:10.314599 16151 raft_consensus.cc:2454] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:10.314719 13185 raft_consensus.cc:2454] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:10.315222 13251 leader_election.cc:302] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:10.315488 19391 raft_consensus.cc:2781] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:10.315560 19391 raft_consensus.cc:482] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:10.315619 19391 raft_consensus.cc:3037] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.318673 19391 raft_consensus.cc:504] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:10.319806 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa973b2d5ead4d7f89a5629c296a2bfd" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:10.320087 13185 raft_consensus.cc:3037] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.323678 13185 raft_consensus.cc:2454] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:10.324231 13251 leader_election.cc:302] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:10.324538 19397 raft_consensus.cc:2781] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:10.324658 19397 raft_consensus.cc:686] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:10.324923 19397 consensus_queue.cc:237] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:10.326567 19391 leader_election.cc:288] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:10.329962 16151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa973b2d5ead4d7f89a5629c296a2bfd" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:10.330192 16151 raft_consensus.cc:3037] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.332870 16151 raft_consensus.cc:2454] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:10.329033 13044 catalog_manager.cc:5484] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:10.340023 19615 tablet_bootstrap.cc:492] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:10.340261 19599 tablet_bootstrap.cc:492] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:10.340365 19615 tablet_bootstrap.cc:492] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap complete.
I20240502 14:15:10.340590 19615 ts_tablet_manager.cc:1367] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.007s
I20240502 14:15:10.340687 19599 tablet_bootstrap.cc:492] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:15:10.340950 19599 ts_tablet_manager.cc:1367] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.082s user 0.048s sys 0.001s
I20240502 14:15:10.341284 19615 raft_consensus.cc:348] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:10.341535 19615 raft_consensus.cc:729] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: LEARNER
I20240502 14:15:10.341670 19599 raft_consensus.cc:348] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:10.341693 19615 consensus_queue.cc:260] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:10.341889 19599 raft_consensus.cc:729] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:15:10.342046 19599 consensus_queue.cc:260] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:10.345485 19615 ts_tablet_manager.cc:1398] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:15:10.346819 19599 ts_tablet_manager.cc:1398] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:15:10.349812 16442 tablet_copy_service.cc:342] P 8b47d4390f8343a493d6c00123785968: Request end of tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-2fc251d4e82e452d989405e121d08dd4 received from {username='slave'} at 127.6.196.195:39073
I20240502 14:15:10.349938 16442 tablet_copy_service.cc:433] P 8b47d4390f8343a493d6c00123785968: ending tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-2fc251d4e82e452d989405e121d08dd4 on tablet 2fc251d4e82e452d989405e121d08dd4 with peer 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:15:10.350886 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-9a95b9049a344b98be91b9961f3e5bee received from {username='slave'} at 127.6.196.197:40779
I20240502 14:15:10.350996 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-9a95b9049a344b98be91b9961f3e5bee on tablet 9a95b9049a344b98be91b9961f3e5bee with peer d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:10.398451 13185 raft_consensus.cc:1261] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:15:10.399672 19443 consensus_queue.cc:1035] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:10.402343 19284 consensus_queue.cc:1035] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.421057 19617 tablet_bootstrap.cc:492] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:10.421715 19617 tablet_bootstrap.cc:492] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:15:10.422047 19617 ts_tablet_manager.cc:1367] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.123s user 0.046s sys 0.008s
I20240502 14:15:10.423482 19617 raft_consensus.cc:348] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:10.423770 19617 raft_consensus.cc:729] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:15:10.423974 19617 consensus_queue.cc:260] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:10.424582 19617 ts_tablet_manager.cc:1398] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:10.433754 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-c3e9160c46d64fa0a964cc39aec91dfd received from {username='slave'} at 127.6.196.197:53727
I20240502 14:15:10.433861 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-c3e9160c46d64fa0a964cc39aec91dfd on tablet c3e9160c46d64fa0a964cc39aec91dfd with peer d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:10.439500 16151 raft_consensus.cc:1261] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240502 14:15:10.441042 19461 consensus_queue.cc:1035] T c1b7405d7ff34356b0aa6ecb12b50817 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.441591 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:10.448722 19413 consensus_queue.cc:1035] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:10.487219 13187 raft_consensus.cc:1261] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:10.490331 19391 consensus_queue.cc:1035] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.490759 16151 raft_consensus.cc:1261] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:10.491430 19391 consensus_queue.cc:1035] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.503785 19664 tablet.cc:1722] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:10.504304 19664 tablet.cc:1620] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: MemRowSet was empty: no flush needed.
I20240502 14:15:10.506958 13044 catalog_manager.cc:5576] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193) does not have the latest schema for tablet aa973b2d5ead4d7f89a5629c296a2bfd (table rebalancer_dynamic_table [id=ccb1096c99524de0b353beec634a3b56]). Expected version 1 got 0
I20240502 14:15:10.510095 13046 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet aa973b2d5ead4d7f89a5629c296a2bfd (table rebalancer_dynamic_table [id=ccb1096c99524de0b353beec634a3b56]). Expected version 1 got 0
I20240502 14:15:10.514317 13046 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet aa973b2d5ead4d7f89a5629c296a2bfd (table rebalancer_dynamic_table [id=ccb1096c99524de0b353beec634a3b56]). Expected version 1 got 0
I20240502 14:15:10.523010 13046 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=ccb1096c99524de0b353beec634a3b56] alter complete (version 1)
I20240502 14:15:10.525553 19665 tablet.cc:1722] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:10.526190 19665 tablet.cc:1620] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:15:10.526919 19666 tablet.cc:1722] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:10.527773 19666 tablet.cc:1620] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:10.530520 13046 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet aa973b2d5ead4d7f89a5629c296a2bfd (table rebalancer_dynamic_table [id=ccb1096c99524de0b353beec634a3b56]). Expected version 1 got 0
I20240502 14:15:10.531726 13044 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet aa973b2d5ead4d7f89a5629c296a2bfd (table rebalancer_dynamic_table [id=ccb1096c99524de0b353beec634a3b56]). Expected version 1 got 0
I20240502 14:15:10.560995 13447 raft_consensus.cc:1203] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:10.576654 13318 consensus_queue.cc:237] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:10.578680 13447 raft_consensus.cc:1261] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:15:10.578739 13252 consensus_peers.cc:485] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ae45cbb45edf41808686de3418231730. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:10.579200 13188 raft_consensus.cc:1261] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:10.579123 19397 consensus_queue.cc:1035] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:10.579634 19391 consensus_queue.cc:1035] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:10.581332 19173 consensus_queue.cc:1035] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.584551 19397 raft_consensus.cc:2932] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:10.585992 13447 raft_consensus.cc:2932] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:10.586179 13188 raft_consensus.cc:2932] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:10.590327 13046 catalog_manager.cc:5484] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index -1 to 4, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:10.615852 19171 consensus_queue.cc:1035] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.625597 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:10.626809 16023 raft_consensus.cc:1203] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:15:10.645044 19664 tablet.cc:1722] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:10.645490 19664 tablet.cc:1620] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: MemRowSet was empty: no flush needed.
I20240502 14:15:10.645699 19666 tablet.cc:1722] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:10.646106 19666 tablet.cc:1620] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:10.653422 19665 tablet.cc:1722] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:10.653824 19665 tablet.cc:1620] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:15:10.660733 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=ccb1096c99524de0b353beec634a3b56] alter complete (version 2)
I20240502 14:15:10.669637 13188 consensus_queue.cc:237] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:10.672039 13318 raft_consensus.cc:1261] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240502 14:15:10.672183 13122 consensus_peers.cc:485] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 22705e1aee0947d8996575b93efd3f1f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:10.672590 19278 consensus_queue.cc:1035] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:10.673491 13444 raft_consensus.cc:1261] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:10.673952 19278 consensus_queue.cc:1035] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:10.674744 19695 raft_consensus.cc:2932] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:10.675302 13318 raft_consensus.cc:2932] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:10.684281 13046 catalog_manager.cc:5484] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:10.691596 13444 raft_consensus.cc:2932] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:10.741816 16023 raft_consensus.cc:1203] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:15:10.765079 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:10.771363 19564 consensus_queue.cc:1035] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.771646 19664 tablet.cc:1722] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:10.772743 19666 tablet.cc:1722] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:10.780397 19665 tablet.cc:1722] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:10.786123 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=ccb1096c99524de0b353beec634a3b56] alter complete (version 3)
I20240502 14:15:10.790771 19171 consensus_queue.cc:1035] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:10.793779 13044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:10.803946 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 76c85c912ab74c32bfed536ea38d23f6 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=82dbfe603cd54bde8f4e20b802fcca34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.804473 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c85c912ab74c32bfed536ea38d23f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.805043 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 94ef25b3a2cd4bc5a53385045336f45a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=82dbfe603cd54bde8f4e20b802fcca34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.805370 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 76c85c912ab74c32bfed536ea38d23f6 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=82dbfe603cd54bde8f4e20b802fcca34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.805456 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ef25b3a2cd4bc5a53385045336f45a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.805750 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c85c912ab74c32bfed536ea38d23f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.810606 16265 tablet_service.cc:1467] Processing CreateTablet for tablet 94ef25b3a2cd4bc5a53385045336f45a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=82dbfe603cd54bde8f4e20b802fcca34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.810878 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 76c85c912ab74c32bfed536ea38d23f6 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=82dbfe603cd54bde8f4e20b802fcca34]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.811008 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ef25b3a2cd4bc5a53385045336f45a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.810932 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4f5c4167ac4c30af3aacb7f7cd099b (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=82dbfe603cd54bde8f4e20b802fcca34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.811326 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c85c912ab74c32bfed536ea38d23f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.810606 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4f5c4167ac4c30af3aacb7f7cd099b (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=82dbfe603cd54bde8f4e20b802fcca34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.811544 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4f5c4167ac4c30af3aacb7f7cd099b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.811686 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4f5c4167ac4c30af3aacb7f7cd099b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.812747 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 5c4f5c4167ac4c30af3aacb7f7cd099b (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=82dbfe603cd54bde8f4e20b802fcca34]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.813081 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4f5c4167ac4c30af3aacb7f7cd099b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.813545 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 94ef25b3a2cd4bc5a53385045336f45a (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=82dbfe603cd54bde8f4e20b802fcca34]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:10.813843 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ef25b3a2cd4bc5a53385045336f45a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:10.834156 19734 tablet_bootstrap.cc:492] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:10.836004 19734 tablet_bootstrap.cc:654] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.837846 19735 tablet_bootstrap.cc:492] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:10.841054 19735 tablet_bootstrap.cc:654] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.843293 19737 tablet_bootstrap.cc:492] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:10.844872 19736 tablet_bootstrap.cc:492] T 94ef25b3a2cd4bc5a53385045336f45a P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:10.845696 19735 tablet_bootstrap.cc:492] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:10.846468 19735 ts_tablet_manager.cc:1367] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 14:15:10.847184 19737 tablet_bootstrap.cc:654] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.847257 19736 tablet_bootstrap.cc:654] T 94ef25b3a2cd4bc5a53385045336f45a P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.849604 19735 raft_consensus.cc:348] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:10.849959 19735 raft_consensus.cc:374] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.849998 19732 tablet_bootstrap.cc:492] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:10.850066 19735 raft_consensus.cc:729] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.850471 19733 tablet_bootstrap.cc:492] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:10.850358 19735 consensus_queue.cc:260] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:10.852924 19732 tablet_bootstrap.cc:654] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.853319 19733 tablet_bootstrap.cc:654] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.853183 19734 tablet_bootstrap.cc:492] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:10.853704 19734 ts_tablet_manager.cc:1367] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240502 14:15:10.854523 19734 raft_consensus.cc:348] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.854779 19734 raft_consensus.cc:374] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.854899 19734 raft_consensus.cc:729] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.854961 19736 tablet_bootstrap.cc:492] T 94ef25b3a2cd4bc5a53385045336f45a P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:10.855190 19736 ts_tablet_manager.cc:1367] T 94ef25b3a2cd4bc5a53385045336f45a P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240502 14:15:10.855113 19734 consensus_queue.cc:260] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.855579 19734 ts_tablet_manager.cc:1398] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:10.855619 19735 ts_tablet_manager.cc:1398] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:15:10.856688 19734 tablet_bootstrap.cc:492] T 94ef25b3a2cd4bc5a53385045336f45a P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:10.857532 19737 tablet_bootstrap.cc:492] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:10.859033 19734 tablet_bootstrap.cc:654] T 94ef25b3a2cd4bc5a53385045336f45a P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.859212 19737 ts_tablet_manager.cc:1367] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240502 14:15:10.860029 19732 tablet_bootstrap.cc:492] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:10.860242 19732 ts_tablet_manager.cc:1367] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240502 14:15:10.860301 19736 raft_consensus.cc:348] T 94ef25b3a2cd4bc5a53385045336f45a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:10.860534 19736 raft_consensus.cc:374] T 94ef25b3a2cd4bc5a53385045336f45a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.860636 19736 raft_consensus.cc:729] T 94ef25b3a2cd4bc5a53385045336f45a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.860882 19736 consensus_queue.cc:260] T 94ef25b3a2cd4bc5a53385045336f45a P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:10.861317 19732 raft_consensus.cc:348] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.861601 19732 raft_consensus.cc:374] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.861747 19732 raft_consensus.cc:729] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.861742 19737 raft_consensus.cc:348] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.861971 19737 raft_consensus.cc:374] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.862105 19737 raft_consensus.cc:729] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.862061 19732 consensus_queue.cc:260] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.862376 19737 consensus_queue.cc:260] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.862625 19732 ts_tablet_manager.cc:1398] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:10.864478 19736 ts_tablet_manager.cc:1398] T 94ef25b3a2cd4bc5a53385045336f45a P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240502 14:15:10.864738 19736 tablet_bootstrap.cc:492] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:10.865804 19736 tablet_bootstrap.cc:654] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.867157 19737 ts_tablet_manager.cc:1398] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240502 14:15:10.870201 13444 consensus_queue.cc:237] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:10.872794 13188 raft_consensus.cc:1261] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:15:10.873283 19413 consensus_queue.cc:1035] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:10.875725 19413 raft_consensus.cc:2932] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:10.880527 13188 raft_consensus.cc:2932] T 6395ce4c30eb4644b0a711de3e60b9c0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:10.883971 16152 raft_consensus.cc:1261] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240502 14:15:10.884490 19736 tablet_bootstrap.cc:492] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:10.884550 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6395ce4c30eb4644b0a711de3e60b9c0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:10.884651 19736 ts_tablet_manager.cc:1367] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.004s
I20240502 14:15:10.884637 19443 consensus_queue.cc:1035] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:10.885490 19736 raft_consensus.cc:348] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.885695 19736 raft_consensus.cc:374] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.885785 19736 raft_consensus.cc:729] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.885857 16152 raft_consensus.cc:2932] T 6395ce4c30eb4644b0a711de3e60b9c0 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:10.886024 19736 consensus_queue.cc:260] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.889276 13044 catalog_manager.cc:5484] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 5 to 7, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New cstate: current_term: 2 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:10.890062 13042 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:10.890266 13042 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:10.894734 13042 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet aa973b2d5ead4d7f89a5629c296a2bfd
I20240502 14:15:10.895637 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet 6395ce4c30eb4644b0a711de3e60b9c0 with delete_type TABLET_DATA_TOMBSTONED (TS cd67493f6f0142f896010f2b8a39887d not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57492
I20240502 14:15:10.895784 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet aa973b2d5ead4d7f89a5629c296a2bfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:10 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:10.896046 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet aa973b2d5ead4d7f89a5629c296a2bfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:10 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:15:10.899729 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet aa973b2d5ead4d7f89a5629c296a2bfd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:10 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:15:10.904346 19747 tablet_replica.cc:330] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:10.904824 19748 tablet_replica.cc:330] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:15:10.905479 19748 raft_consensus.cc:2229] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:10.905961 19747 raft_consensus.cc:2229] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:10.906209 19748 raft_consensus.cc:2258] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:10.906360 19734 tablet_bootstrap.cc:492] T 94ef25b3a2cd4bc5a53385045336f45a P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:10.906579 19734 ts_tablet_manager.cc:1367] T 94ef25b3a2cd4bc5a53385045336f45a P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.014s
I20240502 14:15:10.906924 19747 raft_consensus.cc:2258] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:10.907476 19734 raft_consensus.cc:348] T 94ef25b3a2cd4bc5a53385045336f45a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:10.907778 19734 raft_consensus.cc:374] T 94ef25b3a2cd4bc5a53385045336f45a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.908001 19734 raft_consensus.cc:729] T 94ef25b3a2cd4bc5a53385045336f45a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.908335 19734 consensus_queue.cc:260] T 94ef25b3a2cd4bc5a53385045336f45a P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:10.908923 19734 ts_tablet_manager.cc:1398] T 94ef25b3a2cd4bc5a53385045336f45a P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:10.909947 19748 ts_tablet_manager.cc:1875] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:10.911032 19747 ts_tablet_manager.cc:1875] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:10.913067 19736 ts_tablet_manager.cc:1398] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.028s user 0.001s sys 0.000s
I20240502 14:15:10.919597 19733 tablet_bootstrap.cc:492] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:10.919741 19733 ts_tablet_manager.cc:1367] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.070s user 0.007s sys 0.004s
I20240502 14:15:10.920447 19733 raft_consensus.cc:348] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.920676 19733 raft_consensus.cc:374] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.920776 19733 raft_consensus.cc:729] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.921026 19733 consensus_queue.cc:260] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.921713 19733 ts_tablet_manager.cc:1398] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:10.922070 19733 tablet_bootstrap.cc:492] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:10.923439 19733 tablet_bootstrap.cc:654] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:10.934006 19733 tablet_bootstrap.cc:492] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:10.934147 19733 ts_tablet_manager.cc:1367] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20240502 14:15:10.934594 19733 raft_consensus.cc:348] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.934722 19733 raft_consensus.cc:374] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:10.934787 19733 raft_consensus.cc:729] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:10.934959 19733 consensus_queue.cc:260] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.935366 19733 ts_tablet_manager.cc:1398] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240502 14:15:10.936059 19397 raft_consensus.cc:482] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:10.936173 19397 raft_consensus.cc:504] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.936663 19397 leader_election.cc:288] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:10.937770 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c85c912ab74c32bfed536ea38d23f6" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:10.938040 16420 raft_consensus.cc:2454] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:10.938339 13252 leader_election.cc:302] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:10.938558 19397 raft_consensus.cc:2781] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:10.938616 19397 raft_consensus.cc:482] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:10.938663 19397 raft_consensus.cc:3037] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.938616 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c85c912ab74c32bfed536ea38d23f6" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:10.941130 19397 raft_consensus.cc:504] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.941726 19397 leader_election.cc:288] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:10.942175 16152 raft_consensus.cc:2454] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:10.942229 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c85c912ab74c32bfed536ea38d23f6" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:10.942351 16418 raft_consensus.cc:3037] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.943929 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c85c912ab74c32bfed536ea38d23f6" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:10.944114 16152 raft_consensus.cc:3037] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.944221 16418 raft_consensus.cc:2454] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:10.944500 13252 leader_election.cc:302] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:10.944816 19397 raft_consensus.cc:2781] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:10.944916 19397 raft_consensus.cc:686] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:10.945089 19397 consensus_queue.cc:237] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.946729 16152 raft_consensus.cc:2454] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:10.950045 19746 tablet_replica.cc:330] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:15:10.950379 19748 ts_tablet_manager.cc:1888] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:10.950500 19748 log.cc:1196] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/aa973b2d5ead4d7f89a5629c296a2bfd
I20240502 14:15:10.950856 19748 ts_tablet_manager.cc:1909] T aa973b2d5ead4d7f89a5629c296a2bfd P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:15:10.951757 19747 ts_tablet_manager.cc:1888] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:10.951910 19747 log.cc:1196] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/aa973b2d5ead4d7f89a5629c296a2bfd
I20240502 14:15:10.952116 19391 raft_consensus.cc:482] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:10.952145 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet aa973b2d5ead4d7f89a5629c296a2bfd (table rebalancer_dynamic_table_new_name [id=ccb1096c99524de0b353beec634a3b56]) successfully deleted
I20240502 14:15:10.952224 19391 raft_consensus.cc:504] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.952750 19391 leader_election.cc:288] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:10.953325 19747 ts_tablet_manager.cc:1909] T aa973b2d5ead4d7f89a5629c296a2bfd P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:15:10.953310 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4f5c4167ac4c30af3aacb7f7cd099b" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:10.953490 13580 raft_consensus.cc:2454] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:10.954397 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4f5c4167ac4c30af3aacb7f7cd099b" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:10.954632 16288 raft_consensus.cc:2454] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:10.955338 13251 leader_election.cc:302] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:10.955444 13042 catalog_manager.cc:5484] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:10.956001 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet aa973b2d5ead4d7f89a5629c296a2bfd (table rebalancer_dynamic_table_new_name [id=ccb1096c99524de0b353beec634a3b56]) successfully deleted
I20240502 14:15:10.957074 19397 raft_consensus.cc:2781] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:10.957139 19397 raft_consensus.cc:482] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:10.957182 19397 raft_consensus.cc:3037] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.959481 19397 raft_consensus.cc:504] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.959991 19397 leader_election.cc:288] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:10.960242 19746 raft_consensus.cc:2229] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:10.960430 19746 raft_consensus.cc:2258] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:10.961138 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4f5c4167ac4c30af3aacb7f7cd099b" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:10.961381 13578 raft_consensus.cc:3037] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.962240 19746 ts_tablet_manager.cc:1875] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:10.963516 13578 raft_consensus.cc:2454] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:10.964560 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c4f5c4167ac4c30af3aacb7f7cd099b" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:10.964742 16288 raft_consensus.cc:3037] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:10.967054 16288 raft_consensus.cc:2454] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:10.967932 13253 leader_election.cc:302] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:10.968807 19391 raft_consensus.cc:2781] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:10.972604 19746 ts_tablet_manager.cc:1888] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240502 14:15:10.972743 19746 log.cc:1196] T 6395ce4c30eb4644b0a711de3e60b9c0 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/6395ce4c30eb4644b0a711de3e60b9c0
I20240502 14:15:10.973440 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 6395ce4c30eb4644b0a711de3e60b9c0 (table rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]) successfully deleted
I20240502 14:15:10.973732 19746 tablet_replica.cc:330] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:15:10.974131 19746 raft_consensus.cc:2229] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:10.974489 19746 raft_consensus.cc:2258] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:10.975258 19746 ts_tablet_manager.cc:1875] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:10.975435 19391 raft_consensus.cc:686] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:10.975656 19391 consensus_queue.cc:237] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:10.979476 13042 catalog_manager.cc:5484] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:10.986613 19746 ts_tablet_manager.cc:1888] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:10.986752 19746 log.cc:1196] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/aa973b2d5ead4d7f89a5629c296a2bfd
I20240502 14:15:10.986949 19758 ts_tablet_manager.cc:902] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:10.987221 19746 ts_tablet_manager.cc:1909] T aa973b2d5ead4d7f89a5629c296a2bfd P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:15:10.993105 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet aa973b2d5ead4d7f89a5629c296a2bfd (table rebalancer_dynamic_table_new_name [id=ccb1096c99524de0b353beec634a3b56]) successfully deleted
I20240502 14:15:10.997861 19654 raft_consensus.cc:1050] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968: attempting to promote NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 to VOTER
I20240502 14:15:10.998654 19654 consensus_queue.cc:237] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:11.000118 19758 tablet_copy_client.cc:308] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:15:11.001121 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet ae45cbb45edf41808686de3418231730 from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:56671)
I20240502 14:15:11.001271 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet ae45cbb45edf41808686de3418231730 from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:56671: session id = 8b47d4390f8343a493d6c00123785968-ae45cbb45edf41808686de3418231730
I20240502 14:15:11.002437 13444 raft_consensus.cc:1261] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:11.002663 16152 raft_consensus.cc:1261] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:11.003007 16288 raft_consensus.cc:1261] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:11.003288 19654 consensus_queue.cc:1035] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:11.003684 13339 tablet_copy_source_session.cc:205] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:11.003901 19461 consensus_queue.cc:1035] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:11.005344 13044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:11.005553 19461 consensus_queue.cc:1035] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:11.008648 19758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae45cbb45edf41808686de3418231730. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.020617 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 92a17461403344f791372f28e4756ff5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c92e18f6046f44b2817e965ff29dfbcf]), partition=
I20240502 14:15:11.020859 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 92a17461403344f791372f28e4756ff5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c92e18f6046f44b2817e965ff29dfbcf]), partition=
I20240502 14:15:11.021076 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92a17461403344f791372f28e4756ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.021203 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92a17461403344f791372f28e4756ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.023296 19413 raft_consensus.cc:482] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:11.023433 19413 raft_consensus.cc:504] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:11.024266 19413 leader_election.cc:288] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:11.024958 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ef25b3a2cd4bc5a53385045336f45a" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:11.025178 16287 raft_consensus.cc:2454] T 94ef25b3a2cd4bc5a53385045336f45a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:15:11.025743 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 92a17461403344f791372f28e4756ff5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c92e18f6046f44b2817e965ff29dfbcf]), partition=
I20240502 14:15:11.026026 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92a17461403344f791372f28e4756ff5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.026096 13384 leader_election.cc:302] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:11.026491 19413 raft_consensus.cc:2781] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:11.026568 19413 raft_consensus.cc:482] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:11.026633 19413 raft_consensus.cc:3037] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.026923 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ef25b3a2cd4bc5a53385045336f45a" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:11.027122 13580 raft_consensus.cc:2454] T 94ef25b3a2cd4bc5a53385045336f45a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:15:11.027793 19734 tablet_bootstrap.cc:492] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:11.029287 19734 tablet_bootstrap.cc:654] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.029750 19413 raft_consensus.cc:504] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:11.030948 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ef25b3a2cd4bc5a53385045336f45a" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:11.031175 16287 raft_consensus.cc:3037] T 94ef25b3a2cd4bc5a53385045336f45a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.035145 16287 raft_consensus.cc:2454] T 94ef25b3a2cd4bc5a53385045336f45a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:11.035356 19736 tablet_bootstrap.cc:492] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:11.035550 13384 leader_election.cc:302] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:11.036856 19736 tablet_bootstrap.cc:654] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.036813 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ef25b3a2cd4bc5a53385045336f45a" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:11.037030 13580 raft_consensus.cc:3037] T 94ef25b3a2cd4bc5a53385045336f45a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.037302 19443 raft_consensus.cc:2781] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:11.037407 19443 raft_consensus.cc:686] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:11.037645 19443 consensus_queue.cc:237] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:11.041970 13580 raft_consensus.cc:2454] T 94ef25b3a2cd4bc5a53385045336f45a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:11.042783 19413 leader_election.cc:288] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:11.043570 19758 tablet_copy_client.cc:783] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:11.043922 19758 tablet_copy_client.cc:647] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:11.046120 19758 tablet_copy_client.cc:523] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:11.047621 19654 raft_consensus.cc:2932] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:11.049724 19758 tablet_bootstrap.cc:492] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:11.053361 16155 raft_consensus.cc:2932] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:11.056993 19732 tablet_bootstrap.cc:492] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:11.060679 16288 raft_consensus.cc:2932] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:11.063656 13044 catalog_manager.cc:5484] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:11.068518 16418 consensus_queue.cc:237] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:11.068356 13444 raft_consensus.cc:2932] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:11.073523 19732 tablet_bootstrap.cc:654] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.074702 16155 raft_consensus.cc:1261] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:11.074991 13444 raft_consensus.cc:1261] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:11.075713 19461 consensus_queue.cc:1035] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:11.077210 19461 consensus_queue.cc:1035] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:11.080593 19654 raft_consensus.cc:2932] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:11.080955 13447 raft_consensus.cc:2932] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:11.083554 19736 tablet_bootstrap.cc:492] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:11.083676 19736 ts_tablet_manager.cc:1367] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.005s
I20240502 14:15:11.084578 19736 raft_consensus.cc:348] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:11.084807 19736 raft_consensus.cc:374] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.084873 19736 raft_consensus.cc:729] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.085094 19736 consensus_queue.cc:260] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:11.086808 19732 tablet_bootstrap.cc:492] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:11.087028 19732 ts_tablet_manager.cc:1367] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.003s
I20240502 14:15:11.087604 16155 raft_consensus.cc:2932] T 2fc251d4e82e452d989405e121d08dd4 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:11.087994 19732 raft_consensus.cc:348] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:11.088243 19732 raft_consensus.cc:374] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.088325 19732 raft_consensus.cc:729] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.088603 19732 consensus_queue.cc:260] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:11.089357 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2fc251d4e82e452d989405e121d08dd4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:11.089919 13046 catalog_manager.cc:5484] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:11.099763 19776 ts_tablet_manager.cc:902] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:11.092664 13042 catalog_manager.cc:5484] T 2fc251d4e82e452d989405e121d08dd4 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:11.102286 19734 tablet_bootstrap.cc:492] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:11.102502 19734 ts_tablet_manager.cc:1367] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.005s
I20240502 14:15:11.103654 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fc251d4e82e452d989405e121d08dd4 with delete_type TABLET_DATA_TOMBSTONED (TS 5297e816f782430d9a01220f0471a882 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:11.107003 19732 ts_tablet_manager.cc:1398] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
I20240502 14:15:11.112071 19734 raft_consensus.cc:348] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:11.112839 19734 raft_consensus.cc:374] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.113188 19734 raft_consensus.cc:729] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.114110 19734 consensus_queue.cc:260] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:11.114764 19736 ts_tablet_manager.cc:1398] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
I20240502 14:15:11.121908 19734 ts_tablet_manager.cc:1398] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 14:15:11.123711 19397 raft_consensus.cc:1050] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER d1da7f11f0604f78934354a992bb0f55 to VOTER
I20240502 14:15:11.124567 19397 consensus_queue.cc:237] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:11.129022 19779 tablet_replica.cc:330] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:11.129372 19779 raft_consensus.cc:2229] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:11.129796 19779 raft_consensus.cc:2258] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:11.132906 19779 ts_tablet_manager.cc:1875] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:11.137857 16023 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:11.138408 19391 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:11.139663 13447 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:11.140209 19752 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:11.144764 19779 ts_tablet_manager.cc:1888] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:11.149371 19779 log.cc:1196] T 2fc251d4e82e452d989405e121d08dd4 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/2fc251d4e82e452d989405e121d08dd4
I20240502 14:15:11.147405 16287 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:11.150398 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 2fc251d4e82e452d989405e121d08dd4 (table rebalancer_extra_table_16 [id=bd0be61e86064cbca7fde36c32139e6a]) successfully deleted
I20240502 14:15:11.152143 19397 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:11.156875 19776 tablet_copy_client.cc:308] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:36687
I20240502 14:15:11.157908 13208 tablet_copy_service.cc:140] P 20a967bad5504b1e9c6fb720ecdd1625: Received BeginTabletCopySession request for tablet 22705e1aee0947d8996575b93efd3f1f from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:48115)
I20240502 14:15:11.158073 13208 tablet_copy_service.cc:161] P 20a967bad5504b1e9c6fb720ecdd1625: Beginning new tablet copy session on tablet 22705e1aee0947d8996575b93efd3f1f from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:48115: session id = 5297e816f782430d9a01220f0471a882-22705e1aee0947d8996575b93efd3f1f
I20240502 14:15:11.161782 13208 tablet_copy_source_session.cc:205] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:11.164898 19776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22705e1aee0947d8996575b93efd3f1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.170122 19397 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:11.175046 16287 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:11.176188 16020 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:11.181118 13042 catalog_manager.cc:5484] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:11.182696 19776 tablet_copy_client.cc:783] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:11.182926 19776 tablet_copy_client.cc:647] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:11.185808 19776 tablet_copy_client.cc:523] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:11.191090 19776 tablet_bootstrap.cc:492] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:11.193418 19586 raft_consensus.cc:482] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:11.193588 19586 raft_consensus.cc:504] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:11.194590 19586 leader_election.cc:288] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:11.195475 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a17461403344f791372f28e4756ff5" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:11.195216 13447 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:11.195680 13580 raft_consensus.cc:2454] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:11.196264 16219 leader_election.cc:302] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:15:11.196954 19758 tablet_bootstrap.cc:492] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:11.197396 19758 tablet_bootstrap.cc:492] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:15:11.197630 19758 ts_tablet_manager.cc:1367] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.148s user 0.055s sys 0.006s
I20240502 14:15:11.197495 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a17461403344f791372f28e4756ff5" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:11.197757 16155 raft_consensus.cc:2454] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:11.198522 19758 raft_consensus.cc:348] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:11.198776 19758 raft_consensus.cc:729] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:15:11.199005 19758 consensus_queue.cc:260] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:11.202293 19758 ts_tablet_manager.cc:1398] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:11.203441 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-ae45cbb45edf41808686de3418231730 received from {username='slave'} at 127.6.196.200:56671
I20240502 14:15:11.203513 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 8b47d4390f8343a493d6c00123785968-ae45cbb45edf41808686de3418231730 on tablet ae45cbb45edf41808686de3418231730 with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:15:11.205402 13318 consensus_queue.cc:237] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:11.207674 19792 raft_consensus.cc:2781] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:11.207792 19792 raft_consensus.cc:482] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:11.207943 19792 raft_consensus.cc:3037] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.209178 16287 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:11.209353 19155 raft_consensus.cc:1050] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER d1da7f11f0604f78934354a992bb0f55 to VOTER
I20240502 14:15:11.209753 19391 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:11.210110 19155 consensus_queue.cc:237] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:11.211598 16020 raft_consensus.cc:1261] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:11.212088 19391 consensus_queue.cc:1035] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:11.214167 16023 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:11.214439 19391 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:11.218031 16418 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:11.218521 19413 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:11.219126 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9a95b9049a344b98be91b9961f3e5bee with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:11.219318 13318 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:11.219460 19413 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:11.219779 16287 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:11.221405 19413 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:11.223749 13042 catalog_manager.cc:5484] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 8 to 9, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:11.225215 16020 raft_consensus.cc:2932] T 9a95b9049a344b98be91b9961f3e5bee P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:11.228586 19792 raft_consensus.cc:504] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:11.229445 19792 leader_election.cc:288] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:11.229683 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a17461403344f791372f28e4756ff5" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:11.229832 13580 raft_consensus.cc:3037] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.232223 13580 raft_consensus.cc:2454] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:11.233641 16219 leader_election.cc:302] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:15:11.233884 19792 raft_consensus.cc:2781] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:11.234017 19792 raft_consensus.cc:686] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:11.234258 19792 consensus_queue.cc:237] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:11.237864 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 9a95b9049a344b98be91b9961f3e5bee with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:11.267098 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92a17461403344f791372f28e4756ff5" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:11.267325 16155 raft_consensus.cc:3037] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.268584 13044 catalog_manager.cc:5484] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:11.271179 16155 raft_consensus.cc:2454] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:11.290685 19776 tablet_bootstrap.cc:492] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:11.291077 19776 tablet_bootstrap.cc:492] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882: Bootstrap complete.
I20240502 14:15:11.291357 19776 ts_tablet_manager.cc:1367] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.102s user 0.040s sys 0.003s
I20240502 14:15:11.291997 19801 tablet_replica.cc:330] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:11.292171 19801 raft_consensus.cc:2229] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:11.292069 19776 raft_consensus.cc:348] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:11.292300 19776 raft_consensus.cc:729] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: LEARNER
I20240502 14:15:11.292423 19801 raft_consensus.cc:2258] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:11.292471 19776 consensus_queue.cc:260] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:11.292954 19776 ts_tablet_manager.cc:1398] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:11.294428 13208 tablet_copy_service.cc:342] P 20a967bad5504b1e9c6fb720ecdd1625: Request end of tablet copy session 5297e816f782430d9a01220f0471a882-22705e1aee0947d8996575b93efd3f1f received from {username='slave'} at 127.6.196.199:48115
I20240502 14:15:11.294535 13208 tablet_copy_service.cc:433] P 20a967bad5504b1e9c6fb720ecdd1625: ending tablet copy session 5297e816f782430d9a01220f0471a882-22705e1aee0947d8996575b93efd3f1f on tablet 22705e1aee0947d8996575b93efd3f1f with peer 5297e816f782430d9a01220f0471a882
I20240502 14:15:11.297303 19801 ts_tablet_manager.cc:1875] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:11.298209 19413 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:11.303259 16020 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:11.304591 16418 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:11.306355 19801 ts_tablet_manager.cc:1888] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:15:11.306440 19801 log.cc:1196] T 9a95b9049a344b98be91b9961f3e5bee P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/9a95b9049a344b98be91b9961f3e5bee
I20240502 14:15:11.307993 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 9a95b9049a344b98be91b9961f3e5bee (table rebalance_test_table_15 [id=557f9d603bda4eb385d26221f3612c33]) successfully deleted
I20240502 14:15:11.311439 13046 catalog_manager.cc:5484] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:11.316941 13318 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:11.389261 19391 consensus_queue.cc:1035] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:11.399078 16418 raft_consensus.cc:1203] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:11.416129 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:11.417846 19391 consensus_queue.cc:1035] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:11.424904 13580 raft_consensus.cc:1261] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:11.424974 16155 raft_consensus.cc:1261] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:11.425477 19792 consensus_queue.cc:1035] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:11.425880 19798 consensus_queue.cc:1035] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:11.427932 19413 consensus_queue.cc:1035] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:11.443909 19821 tablet.cc:1722] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:11.445566 19821 tablet.cc:1620] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:11.449673 19822 tablet.cc:1722] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:11.450197 19822 tablet.cc:1620] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: MemRowSet was empty: no flush needed.
I20240502 14:15:11.452726 13042 catalog_manager.cc:5576] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817) does not have the latest schema for tablet 92a17461403344f791372f28e4756ff5 (table rebalancer_dynamic_table [id=c92e18f6046f44b2817e965ff29dfbcf]). Expected version 1 got 0
I20240502 14:15:11.471127 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=c92e18f6046f44b2817e965ff29dfbcf] alter complete (version 1)
I20240502 14:15:11.476024 19752 consensus_queue.cc:1035] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:11.480480 19397 consensus_queue.cc:1035] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:11.480862 13046 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 92a17461403344f791372f28e4756ff5 (table rebalancer_dynamic_table [id=c92e18f6046f44b2817e965ff29dfbcf]). Expected version 1 got 0
I20240502 14:15:11.488608 19155 consensus_queue.cc:1035] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:11.512302 13044 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 92a17461403344f791372f28e4756ff5 (table rebalancer_dynamic_table [id=c92e18f6046f44b2817e965ff29dfbcf]). Expected version 1 got 0
I20240502 14:15:11.513684 19827 tablet.cc:1722] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:11.514236 19827 tablet.cc:1620] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:11.551385 16287 raft_consensus.cc:1203] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240502 14:15:11.567209 13444 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c3e9160c46d64fa0a964cc39aec91dfd"
dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51300
I20240502 14:15:11.567343 13444 raft_consensus.cc:595] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:11.582335 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:11.589259 19821 tablet.cc:1722] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:11.589704 19821 tablet.cc:1620] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:11.589767 19822 tablet.cc:1722] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:11.590178 19822 tablet.cc:1620] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: MemRowSet was empty: no flush needed.
I20240502 14:15:11.591189 19827 tablet.cc:1722] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:11.591789 19827 tablet.cc:1620] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:11.601439 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=c92e18f6046f44b2817e965ff29dfbcf] alter complete (version 2)
I20240502 14:15:11.636744 13044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:11.644867 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 524a9a50c4e843a1ad20f071c83b5aec (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9327d0e10a5e429f8d0387e95ae533aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.645256 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524a9a50c4e843a1ad20f071c83b5aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.645730 15999 tablet_service.cc:1467] Processing CreateTablet for tablet 524a9a50c4e843a1ad20f071c83b5aec (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9327d0e10a5e429f8d0387e95ae533aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.646015 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524a9a50c4e843a1ad20f071c83b5aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.649492 16000 tablet_service.cc:1467] Processing CreateTablet for tablet fae5b25ec9694cea87381d27f4d46384 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9327d0e10a5e429f8d0387e95ae533aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.649524 13427 tablet_service.cc:1467] Processing CreateTablet for tablet fae5b25ec9694cea87381d27f4d46384 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9327d0e10a5e429f8d0387e95ae533aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.649755 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae5b25ec9694cea87381d27f4d46384. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.649840 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae5b25ec9694cea87381d27f4d46384. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.654148 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 9a42ded434514cf28e578eb0046dc5f5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9327d0e10a5e429f8d0387e95ae533aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.654500 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a42ded434514cf28e578eb0046dc5f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.654657 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 524a9a50c4e843a1ad20f071c83b5aec (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9327d0e10a5e429f8d0387e95ae533aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.655067 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524a9a50c4e843a1ad20f071c83b5aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.657611 13166 tablet_service.cc:1467] Processing CreateTablet for tablet fae5b25ec9694cea87381d27f4d46384 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9327d0e10a5e429f8d0387e95ae533aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.657979 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae5b25ec9694cea87381d27f4d46384. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.665560 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 9a42ded434514cf28e578eb0046dc5f5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9327d0e10a5e429f8d0387e95ae533aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.666078 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a42ded434514cf28e578eb0046dc5f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.666441 19865 tablet_bootstrap.cc:492] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:11.667649 19868 tablet_bootstrap.cc:492] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:11.667819 19865 tablet_bootstrap.cc:654] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.669581 19868 tablet_bootstrap.cc:654] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.670006 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 9a42ded434514cf28e578eb0046dc5f5 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=9327d0e10a5e429f8d0387e95ae533aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:11.670449 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a42ded434514cf28e578eb0046dc5f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.691584 19872 tablet_bootstrap.cc:492] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:11.692970 19866 tablet_bootstrap.cc:492] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:11.693995 19865 tablet_bootstrap.cc:492] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:11.694202 19865 ts_tablet_manager.cc:1367] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.016s
I20240502 14:15:11.695029 19872 tablet_bootstrap.cc:654] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.695282 19865 raft_consensus.cc:348] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:11.695562 19865 raft_consensus.cc:374] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.695690 19865 raft_consensus.cc:729] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.695886 19866 tablet_bootstrap.cc:654] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.696170 19865 consensus_queue.cc:260] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:11.698239 19865 ts_tablet_manager.cc:1398] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:15:11.698944 19865 tablet_bootstrap.cc:492] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:11.700728 19865 tablet_bootstrap.cc:654] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.702801 19872 tablet_bootstrap.cc:492] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:11.703330 19872 ts_tablet_manager.cc:1367] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240502 14:15:11.706830 19872 raft_consensus.cc:348] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:11.706969 13044 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:11.708108 19872 raft_consensus.cc:374] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.708496 19872 raft_consensus.cc:729] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.709005 19872 consensus_queue.cc:260] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:11.710040 19872 ts_tablet_manager.cc:1398] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:15:11.718371 19865 tablet_bootstrap.cc:492] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:11.718616 19865 ts_tablet_manager.cc:1367] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.010s
I20240502 14:15:11.719414 19865 raft_consensus.cc:348] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:11.719729 19865 raft_consensus.cc:374] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.719866 19865 raft_consensus.cc:729] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.720280 19865 consensus_queue.cc:260] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:11.728060 19873 tablet_bootstrap.cc:492] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:11.728130 19865 ts_tablet_manager.cc:1398] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240502 14:15:11.728426 19821 tablet.cc:1722] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:11.730866 19873 tablet_bootstrap.cc:654] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.732044 19869 tablet_bootstrap.cc:492] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:11.733927 19822 tablet.cc:1722] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:11.738324 19827 tablet.cc:1722] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:11.740739 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=c92e18f6046f44b2817e965ff29dfbcf] alter complete (version 3)
I20240502 14:15:11.742195 19873 tablet_bootstrap.cc:492] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:11.743450 19869 tablet_bootstrap.cc:654] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.743659 19873 ts_tablet_manager.cc:1367] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.000s
I20240502 14:15:11.751364 19873 raft_consensus.cc:348] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:11.752517 19873 raft_consensus.cc:374] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.752650 19873 raft_consensus.cc:729] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.752966 19873 consensus_queue.cc:260] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:11.757974 19873 ts_tablet_manager.cc:1398] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 14:15:11.758016 19868 tablet_bootstrap.cc:492] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:11.758097 19866 tablet_bootstrap.cc:492] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:11.758162 19868 ts_tablet_manager.cc:1367] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.091s user 0.045s sys 0.026s
I20240502 14:15:11.758224 19866 ts_tablet_manager.cc:1367] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.065s user 0.014s sys 0.015s
I20240502 14:15:11.758927 19866 raft_consensus.cc:348] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:11.759089 19869 tablet_bootstrap.cc:492] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:11.759112 19866 raft_consensus.cc:374] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.759184 19866 raft_consensus.cc:729] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.759009 19868 raft_consensus.cc:348] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:11.759351 19868 raft_consensus.cc:374] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.759371 19866 consensus_queue.cc:260] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:11.759569 19868 raft_consensus.cc:729] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.759826 19866 ts_tablet_manager.cc:1398] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:11.759860 19868 consensus_queue.cc:260] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:11.760107 19866 tablet_bootstrap.cc:492] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:11.760826 19868 ts_tablet_manager.cc:1398] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:11.762703 19866 tablet_bootstrap.cc:654] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.763456 19869 ts_tablet_manager.cc:1367] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.001s
I20240502 14:15:11.763970 19869 raft_consensus.cc:348] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:11.764103 19869 raft_consensus.cc:374] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.764166 19869 raft_consensus.cc:729] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.764400 19869 consensus_queue.cc:260] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:11.764940 19869 ts_tablet_manager.cc:1398] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:15:11.769086 19774 raft_consensus.cc:979] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: : Instructing follower 8b47d4390f8343a493d6c00123785968 to start an election
I20240502 14:15:11.769214 19774 raft_consensus.cc:1067] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Signalling peer 8b47d4390f8343a493d6c00123785968 to start an election
I20240502 14:15:11.769564 19869 tablet_bootstrap.cc:492] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:11.771005 19869 tablet_bootstrap.cc:654] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.775946 19869 tablet_bootstrap.cc:492] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:11.776259 19869 ts_tablet_manager.cc:1367] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240502 14:15:11.776476 19866 tablet_bootstrap.cc:492] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:11.776580 19866 ts_tablet_manager.cc:1367] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20240502 14:15:11.777181 19866 raft_consensus.cc:348] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:11.777367 19866 raft_consensus.cc:374] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.777433 19866 raft_consensus.cc:729] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.777669 19866 consensus_queue.cc:260] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:11.778264 19869 raft_consensus.cc:348] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:11.778424 19869 raft_consensus.cc:374] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.778502 19869 raft_consensus.cc:729] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.778681 19869 consensus_queue.cc:260] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:11.781852 19866 ts_tablet_manager.cc:1398] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:11.782248 19869 ts_tablet_manager.cc:1398] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 14:15:11.785725 16418 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c3e9160c46d64fa0a964cc39aec91dfd"
dest_uuid: "8b47d4390f8343a493d6c00123785968"
from {username='slave'} at 127.6.196.195:39073
I20240502 14:15:11.786015 16418 raft_consensus.cc:482] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:11.786160 16418 raft_consensus.cc:3037] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:11.789541 16418 raft_consensus.cc:504] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:11.790757 16418 leader_election.cc:288] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 2 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:11.790890 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e9160c46d64fa0a964cc39aec91dfd" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:11.791065 13318 raft_consensus.cc:3037] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:11.793015 19397 raft_consensus.cc:1050] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:15:11.793251 13318 raft_consensus.cc:2454] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 2.
I20240502 14:15:11.793383 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e9160c46d64fa0a964cc39aec91dfd" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:11.793515 13444 raft_consensus.cc:3032] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:11.793553 13444 raft_consensus.cc:729] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:11.793718 13444 consensus_queue.cc:260] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:11.793725 19397 consensus_queue.cc:237] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:11.793999 13444 raft_consensus.cc:3037] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:11.796200 16418 raft_consensus.cc:1261] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:11.796200 13444 raft_consensus.cc:2454] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 2.
I20240502 14:15:11.796674 13447 raft_consensus.cc:1261] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:11.796694 19752 consensus_queue.cc:1035] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:11.797068 13188 raft_consensus.cc:1261] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:11.798792 19412 consensus_queue.cc:1035] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:11.800040 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e9160c46d64fa0a964cc39aec91dfd" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:11.800235 16023 raft_consensus.cc:3037] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:11.802785 19412 consensus_queue.cc:1035] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:11.802841 16353 leader_election.cc:302] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, 8b47d4390f8343a493d6c00123785968, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:11.802606 16023 raft_consensus.cc:2454] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 2.
I20240502 14:15:11.804121 19461 raft_consensus.cc:2781] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:11.822784 19461 raft_consensus.cc:686] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 2 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:15:11.823156 19461 consensus_queue.cc:237] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:11.823961 19391 raft_consensus.cc:2932] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:11.824427 13447 raft_consensus.cc:2932] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:11.828018 13188 raft_consensus.cc:2932] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:11.830237 13044 catalog_manager.cc:5484] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 1 to 2, leader changed from 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 2 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:11.830457 13042 catalog_manager.cc:5484] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:11.843632 16420 raft_consensus.cc:2932] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:11.845307 13046 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:11.845506 13046 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:11.848351 13046 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 92a17461403344f791372f28e4756ff5
I20240502 14:15:11.849676 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a17461403344f791372f28e4756ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:11 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:11.849817 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a17461403344f791372f28e4756ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:11 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:11.850390 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet 92a17461403344f791372f28e4756ff5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:11 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:15:11.853068 13318 consensus_queue.cc:237] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:11.855283 13187 raft_consensus.cc:1261] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:11.855688 16420 raft_consensus.cc:1261] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:11.855847 19391 consensus_queue.cc:1035] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:11.856375 19412 consensus_queue.cc:1035] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:11.858170 19412 raft_consensus.cc:2932] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:11.860584 16418 raft_consensus.cc:2932] T ae45cbb45edf41808686de3418231730 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:11.863315 19897 tablet_replica.cc:330] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:11.863078 13187 raft_consensus.cc:2932] T ae45cbb45edf41808686de3418231730 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:11.863515 19897 raft_consensus.cc:2229] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:11.863797 19897 raft_consensus.cc:2258] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:11.864290 19898 tablet_replica.cc:330] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:11.869355 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ae45cbb45edf41808686de3418231730 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:11.870594 13044 catalog_manager.cc:5484] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:11.873773 19564 raft_consensus.cc:482] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:11.873926 19564 raft_consensus.cc:504] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:11.874635 19564 leader_election.cc:288] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:11.876446 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet ae45cbb45edf41808686de3418231730 with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:11.876567 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae5b25ec9694cea87381d27f4d46384" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:11.876737 13447 raft_consensus.cc:2454] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:11.877172 15955 leader_election.cc:302] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:11.877458 19171 raft_consensus.cc:2781] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:11.877542 19171 raft_consensus.cc:482] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:11.877635 19171 raft_consensus.cc:3037] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.878813 19899 tablet_replica.cc:330] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:15:11.879050 19899 raft_consensus.cc:2229] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:11.879225 19899 raft_consensus.cc:2258] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:11.880004 19171 raft_consensus.cc:504] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:11.880563 19171 leader_election.cc:288] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:11.881825 19899 ts_tablet_manager.cc:1875] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:11.882104 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae5b25ec9694cea87381d27f4d46384" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:11.882268 13447 raft_consensus.cc:3037] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.882313 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae5b25ec9694cea87381d27f4d46384" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:11.882467 13188 raft_consensus.cc:3037] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.883540 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fae5b25ec9694cea87381d27f4d46384" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:11.884533 13447 raft_consensus.cc:2454] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:11.884582 13188 raft_consensus.cc:2454] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:11.885341 15955 leader_election.cc:302] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:11.885845 19564 raft_consensus.cc:2781] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:11.885907 19898 raft_consensus.cc:2229] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:11.885979 19564 raft_consensus.cc:686] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:11.886149 19898 raft_consensus.cc:2258] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:11.886190 19564 consensus_queue.cc:237] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:11.886720 19897 ts_tablet_manager.cc:1875] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:11.889451 19898 ts_tablet_manager.cc:1875] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:11.897547 19899 ts_tablet_manager.cc:1888] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:11.897689 19899 log.cc:1196] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/92a17461403344f791372f28e4756ff5
I20240502 14:15:11.899581 19899 ts_tablet_manager.cc:1909] T 92a17461403344f791372f28e4756ff5 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:15:11.900126 13044 catalog_manager.cc:5484] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:11.900568 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 92a17461403344f791372f28e4756ff5 (table rebalancer_dynamic_table_new_name [id=c92e18f6046f44b2817e965ff29dfbcf]) successfully deleted
I20240502 14:15:11.905856 19792 raft_consensus.cc:482] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:11.906122 19792 raft_consensus.cc:504] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:11.906733 19792 leader_election.cc:288] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:11.907889 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a42ded434514cf28e578eb0046dc5f5" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:11.908053 13447 raft_consensus.cc:2454] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:11.908428 19900 tablet_replica.cc:330] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:11.908414 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a42ded434514cf28e578eb0046dc5f5" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:11.908587 13318 raft_consensus.cc:2454] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:11.908989 16221 leader_election.cc:302] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:11.909335 19900 raft_consensus.cc:2229] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:11.909570 19792 raft_consensus.cc:2781] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:11.909648 19792 raft_consensus.cc:482] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:11.909726 19792 raft_consensus.cc:3037] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.911062 19900 raft_consensus.cc:2258] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:11.912248 19792 raft_consensus.cc:504] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:11.913062 19900 ts_tablet_manager.cc:1875] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:11.913727 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a42ded434514cf28e578eb0046dc5f5" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:11.913873 13447 raft_consensus.cc:3037] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.916208 13447 raft_consensus.cc:2454] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:11.916668 16219 leader_election.cc:302] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:11.916900 19586 raft_consensus.cc:2781] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:11.917022 19586 raft_consensus.cc:686] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:11.917572 19586 consensus_queue.cc:237] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:11.918855 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a42ded434514cf28e578eb0046dc5f5" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:11.919021 13318 raft_consensus.cc:3037] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:11.922004 13318 raft_consensus.cc:2454] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:11.928845 19900 ts_tablet_manager.cc:1888] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:11.928984 19900 log.cc:1196] T ae45cbb45edf41808686de3418231730 P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/ae45cbb45edf41808686de3418231730
I20240502 14:15:11.931295 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet ae45cbb45edf41808686de3418231730 (table rebalance_test_table_7 [id=fe675f309a11456aaf1fc329cfdb1707]) successfully deleted
I20240502 14:15:11.932797 13044 catalog_manager.cc:5484] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:11.938045 19792 leader_election.cc:288] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:11.941577 19897 ts_tablet_manager.cc:1888] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:11.941680 19897 log.cc:1196] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/92a17461403344f791372f28e4756ff5
I20240502 14:15:11.943259 19897 ts_tablet_manager.cc:1909] T 92a17461403344f791372f28e4756ff5 P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:15:11.943997 19898 ts_tablet_manager.cc:1888] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:11.944582 19898 log.cc:1196] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/92a17461403344f791372f28e4756ff5
I20240502 14:15:11.944665 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 92a17461403344f791372f28e4756ff5 (table rebalancer_dynamic_table_new_name [id=c92e18f6046f44b2817e965ff29dfbcf]) successfully deleted
I20240502 14:15:11.945019 19898 ts_tablet_manager.cc:1909] T 92a17461403344f791372f28e4756ff5 P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:15:11.945915 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 92a17461403344f791372f28e4756ff5 (table rebalancer_dynamic_table_new_name [id=c92e18f6046f44b2817e965ff29dfbcf]) successfully deleted
I20240502 14:15:11.951169 13044 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:11.958037 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 25ca9d8a67ea4e72993e710687e623d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a3565e94b3a6436d8daf1f14a60f7afe]), partition=
I20240502 14:15:11.958217 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 25ca9d8a67ea4e72993e710687e623d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a3565e94b3a6436d8daf1f14a60f7afe]), partition=
I20240502 14:15:11.958303 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ca9d8a67ea4e72993e710687e623d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.958546 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ca9d8a67ea4e72993e710687e623d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.961742 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 25ca9d8a67ea4e72993e710687e623d7 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a3565e94b3a6436d8daf1f14a60f7afe]), partition=
I20240502 14:15:11.962052 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25ca9d8a67ea4e72993e710687e623d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:11.962764 19872 tablet_bootstrap.cc:492] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:11.963729 19868 tablet_bootstrap.cc:492] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:11.963784 19872 tablet_bootstrap.cc:654] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.965690 19868 tablet_bootstrap.cc:654] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.966527 19873 tablet_bootstrap.cc:492] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:11.967892 19873 tablet_bootstrap.cc:654] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:11.970566 19872 tablet_bootstrap.cc:492] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:11.970724 19873 tablet_bootstrap.cc:492] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:11.970759 19872 ts_tablet_manager.cc:1367] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240502 14:15:11.970898 19873 ts_tablet_manager.cc:1367] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:11.971611 19873 raft_consensus.cc:348] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:11.971746 19873 raft_consensus.cc:374] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.971835 19873 raft_consensus.cc:729] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.971671 19872 raft_consensus.cc:348] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:11.972088 19873 consensus_queue.cc:260] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:11.972208 19872 raft_consensus.cc:374] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.972322 19872 raft_consensus.cc:729] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.973101 19872 consensus_queue.cc:260] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:11.975880 19873 ts_tablet_manager.cc:1398] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:11.977182 19868 tablet_bootstrap.cc:492] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:11.977357 19868 ts_tablet_manager.cc:1367] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20240502 14:15:11.977985 19868 raft_consensus.cc:348] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:11.978178 19868 raft_consensus.cc:374] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:11.978242 19868 raft_consensus.cc:729] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:11.978421 19868 consensus_queue.cc:260] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:11.978924 19868 ts_tablet_manager.cc:1398] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:15:11.982076 19872 ts_tablet_manager.cc:1398] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240502 14:15:12.061810 19846 raft_consensus.cc:1050] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625: attempting to promote NON_VOTER 5297e816f782430d9a01220f0471a882 to VOTER
I20240502 14:15:12.062417 19846 consensus_queue.cc:237] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:12.064246 13447 raft_consensus.cc:1261] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:12.064324 13318 raft_consensus.cc:1261] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:12.064368 16287 raft_consensus.cc:1261] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:12.064738 19846 consensus_queue.cc:1035] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:12.064980 19847 consensus_queue.cc:1035] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:12.065270 19913 consensus_queue.cc:1035] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:12.068665 19391 raft_consensus.cc:482] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:12.068818 19391 raft_consensus.cc:504] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:12.069368 19391 leader_election.cc:288] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:12.069999 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ca9d8a67ea4e72993e710687e623d7" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:12.070284 16288 raft_consensus.cc:2454] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:12.070263 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ca9d8a67ea4e72993e710687e623d7" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:12.070637 13253 leader_election.cc:302] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:12.070670 16152 raft_consensus.cc:2454] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:12.071400 19391 raft_consensus.cc:2781] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:12.071545 19391 raft_consensus.cc:482] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:12.071664 19391 raft_consensus.cc:3037] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.073788 19391 raft_consensus.cc:504] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:12.076521 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ca9d8a67ea4e72993e710687e623d7" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:12.076719 16152 raft_consensus.cc:3037] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.076715 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25ca9d8a67ea4e72993e710687e623d7" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:12.076866 16288 raft_consensus.cc:3037] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.079205 16152 raft_consensus.cc:2454] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:12.079563 13252 leader_election.cc:302] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:12.079794 19412 raft_consensus.cc:2781] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:12.079988 19412 raft_consensus.cc:686] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:12.080210 19412 consensus_queue.cc:237] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:12.081172 19847 raft_consensus.cc:2932] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:12.081640 16288 raft_consensus.cc:2454] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:12.080790 19391 leader_election.cc:288] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:12.081964 16287 raft_consensus.cc:2932] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:12.085826 13318 raft_consensus.cc:2932] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:12.087666 13046 catalog_manager.cc:5484] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:12.088363 13447 raft_consensus.cc:2932] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:12.090219 13044 catalog_manager.cc:5484] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:12.094877 13188 consensus_queue.cc:237] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:12.097265 16287 raft_consensus.cc:1261] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:12.097728 19847 consensus_queue.cc:1035] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:12.097883 13318 raft_consensus.cc:1261] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:12.098505 19847 consensus_queue.cc:1035] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:12.100733 19846 raft_consensus.cc:2932] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:12.101480 16287 raft_consensus.cc:2932] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:12.104985 13319 raft_consensus.cc:2932] T 22705e1aee0947d8996575b93efd3f1f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:12.109499 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 22705e1aee0947d8996575b93efd3f1f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:12.108662 13044 catalog_manager.cc:5484] T 22705e1aee0947d8996575b93efd3f1f P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 5 to 6, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:12.114085 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 22705e1aee0947d8996575b93efd3f1f with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:12.114387 19900 tablet_replica.cc:330] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:12.114538 19900 raft_consensus.cc:2229] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:12.114714 19900 raft_consensus.cc:2258] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:12.115432 19900 ts_tablet_manager.cc:1875] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:12.121666 19900 ts_tablet_manager.cc:1888] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240502 14:15:12.121768 19900 log.cc:1196] T 22705e1aee0947d8996575b93efd3f1f P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/22705e1aee0947d8996575b93efd3f1f
I20240502 14:15:12.125447 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 22705e1aee0947d8996575b93efd3f1f (table rebalance_test_table_5 [id=c97a45b4ee8c48a38ebd0709da5770f7]) successfully deleted
I20240502 14:15:12.148764 19564 raft_consensus.cc:482] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:12.148924 19564 raft_consensus.cc:504] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:12.149569 19564 leader_election.cc:288] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:12.149823 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524a9a50c4e843a1ad20f071c83b5aec" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:12.150054 13188 raft_consensus.cc:2454] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:12.150312 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524a9a50c4e843a1ad20f071c83b5aec" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:12.150462 15955 leader_election.cc:302] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:12.150580 16155 raft_consensus.cc:2454] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:12.151444 19564 raft_consensus.cc:2781] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:12.151584 19564 raft_consensus.cc:482] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:12.151681 19564 raft_consensus.cc:3037] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.155432 19564 raft_consensus.cc:504] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:12.156387 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524a9a50c4e843a1ad20f071c83b5aec" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:12.156518 13188 raft_consensus.cc:3037] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.156579 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524a9a50c4e843a1ad20f071c83b5aec" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:12.156795 16155 raft_consensus.cc:3037] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.158989 13188 raft_consensus.cc:2454] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:12.161589 15955 leader_election.cc:302] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:12.161765 16155 raft_consensus.cc:2454] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:12.162113 19171 raft_consensus.cc:2781] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:12.161302 19564 leader_election.cc:288] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:12.162302 19171 raft_consensus.cc:686] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:12.162609 19171 consensus_queue.cc:237] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:12.167217 13043 catalog_manager.cc:5484] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:12.213718 13043 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:12.219908 16155 raft_consensus.cc:1261] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:12.220485 16287 raft_consensus.cc:1261] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:12.220480 19412 consensus_queue.cc:1035] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.221122 19412 consensus_queue.cc:1035] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.228363 19827 tablet.cc:1722] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:12.228904 19827 tablet.cc:1620] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:12.229326 19944 tablet.cc:1722] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:12.229846 19944 tablet.cc:1620] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: MemRowSet was empty: no flush needed.
I20240502 14:15:12.231742 13043 catalog_manager.cc:5576] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041) does not have the latest schema for tablet 25ca9d8a67ea4e72993e710687e623d7 (table rebalancer_dynamic_table [id=a3565e94b3a6436d8daf1f14a60f7afe]). Expected version 1 got 0
I20240502 14:15:12.238364 13043 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=a3565e94b3a6436d8daf1f14a60f7afe] alter complete (version 1)
I20240502 14:15:12.243300 19945 tablet.cc:1722] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:12.243909 19945 tablet.cc:1620] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:12.247156 13046 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet 25ca9d8a67ea4e72993e710687e623d7 (table rebalancer_dynamic_table [id=a3565e94b3a6436d8daf1f14a60f7afe]). Expected version 1 got 0
I20240502 14:15:12.300484 16023 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:15:12.301183 19461 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:12.342088 13043 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:12.348623 19944 tablet.cc:1722] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:12.349073 19944 tablet.cc:1620] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: MemRowSet was empty: no flush needed.
I20240502 14:15:12.349712 19945 tablet.cc:1722] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:12.350066 19945 tablet.cc:1620] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:12.350764 19827 tablet.cc:1722] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:12.351240 19827 tablet.cc:1620] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:12.359010 13046 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=a3565e94b3a6436d8daf1f14a60f7afe] alter complete (version 2)
I20240502 14:15:12.379621 13319 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:15:12.380234 19461 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:12.387576 13447 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:15:12.391657 19890 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:12.393885 16420 consensus_queue.cc:237] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:12.395704 13319 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:15:12.396214 19461 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:15:12.397128 16023 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:15:12.397888 19461 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:15:12.399832 19890 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P 8b47d4390f8343a493d6c00123785968 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:12.404413 16023 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:12.409314 13319 raft_consensus.cc:2932] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:12.415704 13046 catalog_manager.cc:5484] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 8 to 10, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 2 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:12.422341 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet c3e9160c46d64fa0a964cc39aec91dfd with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:12.425899 19900 tablet_replica.cc:330] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:12.426110 19900 raft_consensus.cc:2229] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:12.426291 19900 raft_consensus.cc:2258] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:12.433573 19900 ts_tablet_manager.cc:1875] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:12.433815 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c3e9160c46d64fa0a964cc39aec91dfd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:12.434473 19792 consensus_queue.cc:1035] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.442291 19900 ts_tablet_manager.cc:1888] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 14:15:12.442481 19900 log.cc:1196] T c3e9160c46d64fa0a964cc39aec91dfd P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/c3e9160c46d64fa0a964cc39aec91dfd
I20240502 14:15:12.443434 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet c3e9160c46d64fa0a964cc39aec91dfd (table rebalance_test_table_17 [id=cbc2cd67f3c24195825b542cf2878195]) successfully deleted
I20240502 14:15:12.446638 19792 consensus_queue.cc:1035] T 9a42ded434514cf28e578eb0046dc5f5 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.457762 19904 consensus_queue.cc:1035] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.464640 13043 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:12.467854 19904 consensus_queue.cc:1035] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.478551 19944 tablet.cc:1722] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:12.480419 19827 tablet.cc:1722] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:12.488592 19945 tablet.cc:1722] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:12.491158 13043 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=a3565e94b3a6436d8daf1f14a60f7afe] alter complete (version 3)
I20240502 14:15:12.595912 13043 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:12.596138 13043 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:12.606680 13043 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 25ca9d8a67ea4e72993e710687e623d7
I20240502 14:15:12.607614 16266 tablet_service.cc:1514] Processing DeleteTablet for tablet 25ca9d8a67ea4e72993e710687e623d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:12 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:12.608345 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet 25ca9d8a67ea4e72993e710687e623d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:12 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:15:12.608472 19999 tablet_replica.cc:330] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:12.608673 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet 25ca9d8a67ea4e72993e710687e623d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:12 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:12.608944 19999 raft_consensus.cc:2229] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:12.609292 19999 raft_consensus.cc:2258] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:12.611045 19999 ts_tablet_manager.cc:1875] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:12.611208 19564 consensus_queue.cc:1035] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.622694 19999 ts_tablet_manager.cc:1888] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:12.622876 19999 log.cc:1196] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/25ca9d8a67ea4e72993e710687e623d7
I20240502 14:15:12.623387 19999 ts_tablet_manager.cc:1909] T 25ca9d8a67ea4e72993e710687e623d7 P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:15:12.624225 20000 tablet_replica.cc:330] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:15:12.624357 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 25ca9d8a67ea4e72993e710687e623d7 (table rebalancer_dynamic_table_new_name [id=a3565e94b3a6436d8daf1f14a60f7afe]) successfully deleted
I20240502 14:15:12.624804 20000 raft_consensus.cc:2229] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:12.625174 20000 raft_consensus.cc:2258] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:12.627595 19904 consensus_queue.cc:1035] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:12.635915 20001 tablet_replica.cc:330] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:12.636283 20001 raft_consensus.cc:2229] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:12.636620 20001 raft_consensus.cc:2258] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:12.637725 20001 ts_tablet_manager.cc:1875] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:12.641781 20000 ts_tablet_manager.cc:1875] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:12.644767 20001 ts_tablet_manager.cc:1888] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:12.645076 20001 log.cc:1196] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/25ca9d8a67ea4e72993e710687e623d7
I20240502 14:15:12.648505 20001 ts_tablet_manager.cc:1909] T 25ca9d8a67ea4e72993e710687e623d7 P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:15:12.649760 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 25ca9d8a67ea4e72993e710687e623d7 (table rebalancer_dynamic_table_new_name [id=a3565e94b3a6436d8daf1f14a60f7afe]) successfully deleted
I20240502 14:15:12.662367 20000 ts_tablet_manager.cc:1888] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:12.662485 20000 log.cc:1196] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/25ca9d8a67ea4e72993e710687e623d7
I20240502 14:15:12.662927 20000 ts_tablet_manager.cc:1909] T 25ca9d8a67ea4e72993e710687e623d7 P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:15:12.664113 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 25ca9d8a67ea4e72993e710687e623d7 (table rebalancer_dynamic_table_new_name [id=a3565e94b3a6436d8daf1f14a60f7afe]) successfully deleted
I20240502 14:15:12.709568 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:12.719401 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 771a509653534bfb914bf3a92667f35d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ebdb2763bc648089775988f29782a77]), partition=
I20240502 14:15:12.719448 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 771a509653534bfb914bf3a92667f35d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ebdb2763bc648089775988f29782a77]), partition=
I20240502 14:15:12.719767 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771a509653534bfb914bf3a92667f35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.719750 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 771a509653534bfb914bf3a92667f35d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=0ebdb2763bc648089775988f29782a77]), partition=
I20240502 14:15:12.719846 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771a509653534bfb914bf3a92667f35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.720089 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 771a509653534bfb914bf3a92667f35d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.730608 20010 tablet_bootstrap.cc:492] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:12.731159 20009 tablet_bootstrap.cc:492] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:12.733372 20010 tablet_bootstrap.cc:654] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.734246 20009 tablet_bootstrap.cc:654] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.740375 20010 tablet_bootstrap.cc:492] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:12.740597 20010 ts_tablet_manager.cc:1367] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20240502 14:15:12.741658 20010 raft_consensus.cc:348] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.742151 20010 raft_consensus.cc:374] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.742344 20010 raft_consensus.cc:729] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.742599 20010 consensus_queue.cc:260] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.744475 20010 ts_tablet_manager.cc:1398] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:15:12.747864 20011 tablet_bootstrap.cc:492] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:12.750016 20011 tablet_bootstrap.cc:654] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.750447 13168 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240502 14:15:12.753930 20009 tablet_bootstrap.cc:492] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:12.754204 20009 ts_tablet_manager.cc:1367] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.004s
I20240502 14:15:12.755527 20009 raft_consensus.cc:348] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.757151 20009 raft_consensus.cc:374] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.757297 20009 raft_consensus.cc:729] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.757580 20009 consensus_queue.cc:260] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.758335 20009 ts_tablet_manager.cc:1398] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:12.788368 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:12.797626 16266 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240502 14:15:12.801201 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 07012e715ded463db5edbb6019ee69cf (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=27c4da9c38904014b9c3ed306127c739]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.801679 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07012e715ded463db5edbb6019ee69cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.801883 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 94f535ca99d54963a3fcd54f905281e9 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=27c4da9c38904014b9c3ed306127c739]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.802223 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f535ca99d54963a3fcd54f905281e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.803793 16265 tablet_service.cc:1467] Processing CreateTablet for tablet 61c065054cdb456da06172650abf0fae (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=27c4da9c38904014b9c3ed306127c739]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.804152 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c065054cdb456da06172650abf0fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.805541 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 94f535ca99d54963a3fcd54f905281e9 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=27c4da9c38904014b9c3ed306127c739]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.805598 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 07012e715ded463db5edbb6019ee69cf (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=27c4da9c38904014b9c3ed306127c739]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.805915 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f535ca99d54963a3fcd54f905281e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.805963 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07012e715ded463db5edbb6019ee69cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.809149 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 94f535ca99d54963a3fcd54f905281e9 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=27c4da9c38904014b9c3ed306127c739]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.809509 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94f535ca99d54963a3fcd54f905281e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.809473 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 07012e715ded463db5edbb6019ee69cf (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=27c4da9c38904014b9c3ed306127c739]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.809494 16133 tablet_service.cc:1467] Processing CreateTablet for tablet 61c065054cdb456da06172650abf0fae (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=27c4da9c38904014b9c3ed306127c739]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.809808 16133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c065054cdb456da06172650abf0fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.814900 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 61c065054cdb456da06172650abf0fae (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=27c4da9c38904014b9c3ed306127c739]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:12.815312 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c065054cdb456da06172650abf0fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.821825 20009 tablet_bootstrap.cc:492] T 94f535ca99d54963a3fcd54f905281e9 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:12.823216 20009 tablet_bootstrap.cc:654] T 94f535ca99d54963a3fcd54f905281e9 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.825119 20011 tablet_bootstrap.cc:492] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:12.825277 20011 ts_tablet_manager.cc:1367] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.078s user 0.022s sys 0.034s
I20240502 14:15:12.826143 20011 raft_consensus.cc:348] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.826349 20011 raft_consensus.cc:374] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.826474 20011 raft_consensus.cc:729] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.826715 20011 consensus_queue.cc:260] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.827399 20011 ts_tablet_manager.cc:1398] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:12.829056 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07012e715ded463db5edbb6019ee69cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:12.830363 20027 tablet_bootstrap.cc:492] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:12.832623 20027 tablet_bootstrap.cc:654] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.842156 20011 tablet_bootstrap.cc:492] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:12.843653 20011 tablet_bootstrap.cc:654] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.853361 20027 tablet_bootstrap.cc:492] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:12.853576 20027 ts_tablet_manager.cc:1367] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.013s
I20240502 14:15:12.854764 20027 raft_consensus.cc:348] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.855136 20027 raft_consensus.cc:374] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.855288 20027 raft_consensus.cc:729] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.855701 20027 consensus_queue.cc:260] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.857578 20009 tablet_bootstrap.cc:492] T 94f535ca99d54963a3fcd54f905281e9 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:12.857697 20027 ts_tablet_manager.cc:1398] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:12.857712 20009 ts_tablet_manager.cc:1367] T 94f535ca99d54963a3fcd54f905281e9 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.004s
I20240502 14:15:12.858590 20009 raft_consensus.cc:348] T 94f535ca99d54963a3fcd54f905281e9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.858776 20009 raft_consensus.cc:374] T 94f535ca99d54963a3fcd54f905281e9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.858834 20009 raft_consensus.cc:729] T 94f535ca99d54963a3fcd54f905281e9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.859001 20009 consensus_queue.cc:260] T 94f535ca99d54963a3fcd54f905281e9 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.859544 20009 ts_tablet_manager.cc:1398] T 94f535ca99d54963a3fcd54f905281e9 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:12.865880 20025 tablet_bootstrap.cc:492] T 94f535ca99d54963a3fcd54f905281e9 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:12.868184 20025 tablet_bootstrap.cc:654] T 94f535ca99d54963a3fcd54f905281e9 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.875257 19155 raft_consensus.cc:482] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:12.875365 19155 raft_consensus.cc:504] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.875492 20011 tablet_bootstrap.cc:492] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:12.875627 20011 ts_tablet_manager.cc:1367] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.007s
I20240502 14:15:12.875984 19155 leader_election.cc:288] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:12.876379 20011 raft_consensus.cc:348] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:12.876624 20011 raft_consensus.cc:374] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.876684 20011 raft_consensus.cc:729] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.876592 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a509653534bfb914bf3a92667f35d" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:12.876801 13187 raft_consensus.cc:2454] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:15:12.876904 20011 consensus_queue.cc:260] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:12.877558 20011 ts_tablet_manager.cc:1398] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:12.877800 20024 tablet_bootstrap.cc:492] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:12.878093 13382 leader_election.cc:302] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:12.878398 19155 raft_consensus.cc:2781] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:12.878515 19155 raft_consensus.cc:482] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:12.878614 19155 raft_consensus.cc:3037] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.879731 20024 tablet_bootstrap.cc:654] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.881210 19155 raft_consensus.cc:504] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.884191 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a509653534bfb914bf3a92667f35d" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:12.884387 13187 raft_consensus.cc:3037] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.886830 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a509653534bfb914bf3a92667f35d" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:12.887051 16419 raft_consensus.cc:2454] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:15:12.888609 13187 raft_consensus.cc:2454] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:12.890188 20025 tablet_bootstrap.cc:492] T 94f535ca99d54963a3fcd54f905281e9 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:12.890309 20025 ts_tablet_manager.cc:1367] T 94f535ca99d54963a3fcd54f905281e9 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20240502 14:15:12.892271 20025 raft_consensus.cc:348] T 94f535ca99d54963a3fcd54f905281e9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.892436 20025 raft_consensus.cc:374] T 94f535ca99d54963a3fcd54f905281e9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.892474 20025 raft_consensus.cc:729] T 94f535ca99d54963a3fcd54f905281e9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.893081 20025 consensus_queue.cc:260] T 94f535ca99d54963a3fcd54f905281e9 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:12.893491 20024 tablet_bootstrap.cc:492] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:12.893576 20025 ts_tablet_manager.cc:1398] T 94f535ca99d54963a3fcd54f905281e9 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:12.893826 20025 tablet_bootstrap.cc:492] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:12.895175 20025 tablet_bootstrap.cc:654] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.895417 20024 ts_tablet_manager.cc:1367] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20240502 14:15:12.895906 13382 leader_election.cc:302] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:12.896692 19413 raft_consensus.cc:2781] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:12.900197 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "771a509653534bfb914bf3a92667f35d" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:12.900465 16421 raft_consensus.cc:3037] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:12.896690 20024 raft_consensus.cc:348] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:12.901983 20024 raft_consensus.cc:374] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.902074 20024 raft_consensus.cc:729] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.902271 20024 consensus_queue.cc:260] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:12.902837 20024 ts_tablet_manager.cc:1398] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240502 14:15:12.903103 20024 tablet_bootstrap.cc:492] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:12.904649 20024 tablet_bootstrap.cc:654] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.903956 16421 raft_consensus.cc:2454] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:12.906554 19951 raft_consensus.cc:482] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:12.906688 19951 raft_consensus.cc:504] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:12.907419 19951 leader_election.cc:288] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:12.908483 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07012e715ded463db5edbb6019ee69cf" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
W20240502 14:15:12.909361 16354 leader_election.cc:341] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:12.910802 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07012e715ded463db5edbb6019ee69cf" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
W20240502 14:15:12.913066 16355 leader_election.cc:341] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:12.913234 16355 leader_election.cc:302] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b47d4390f8343a493d6c00123785968; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:12.913669 19951 raft_consensus.cc:2726] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:12.922194 19155 leader_election.cc:288] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:12.942622 19413 raft_consensus.cc:686] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:12.943009 19413 consensus_queue.cc:237] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
W20240502 14:15:12.943563 13514 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.193:36687, user_credentials={real_user=slave}} blocked reactor thread for 73824.6us
I20240502 14:15:12.953145 20023 tablet_bootstrap.cc:492] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:12.953183 13040 catalog_manager.cc:5484] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:12.968396 16399 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:12.971097 20023 tablet_bootstrap.cc:654] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:12.993682 20023 tablet_bootstrap.cc:492] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:12.994408 20023 ts_tablet_manager.cc:1367] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20240502 14:15:12.995293 20023 raft_consensus.cc:348] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:12.998608 20023 raft_consensus.cc:374] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:12.998845 20023 raft_consensus.cc:729] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:12.999253 20023 consensus_queue.cc:260] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:13.004038 16135 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:15:13.012307 20023 ts_tablet_manager.cc:1398] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240502 14:15:13.016741 13559 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:15:13.057500 20023 tablet_bootstrap.cc:492] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:13.058869 19171 raft_consensus.cc:482] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:13.059116 19171 raft_consensus.cc:504] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:13.060214 19171 leader_election.cc:288] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:13.060993 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f535ca99d54963a3fcd54f905281e9" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:13.061126 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f535ca99d54963a3fcd54f905281e9" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:13.061318 13444 raft_consensus.cc:2454] T 94f535ca99d54963a3fcd54f905281e9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:13.061381 13580 raft_consensus.cc:2454] T 94f535ca99d54963a3fcd54f905281e9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:13.061961 15955 leader_election.cc:302] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:13.062592 19904 raft_consensus.cc:2781] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:13.062755 19904 raft_consensus.cc:482] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:13.062855 19904 raft_consensus.cc:3037] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.067631 19904 raft_consensus.cc:504] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:13.069586 20023 tablet_bootstrap.cc:654] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.069504 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f535ca99d54963a3fcd54f905281e9" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:13.069700 13578 raft_consensus.cc:3037] T 94f535ca99d54963a3fcd54f905281e9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.074545 13444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94f535ca99d54963a3fcd54f905281e9" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:13.074724 13444 raft_consensus.cc:3037] T 94f535ca99d54963a3fcd54f905281e9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.079478 19904 leader_election.cc:288] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:13.073154 13578 raft_consensus.cc:2454] T 94f535ca99d54963a3fcd54f905281e9 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:13.080183 15955 leader_election.cc:302] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:13.077209 13444 raft_consensus.cc:2454] T 94f535ca99d54963a3fcd54f905281e9 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:13.084264 19904 raft_consensus.cc:2781] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:13.084888 19904 raft_consensus.cc:686] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:13.085255 19904 consensus_queue.cc:237] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:13.088927 20023 tablet_bootstrap.cc:492] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:13.089080 20023 ts_tablet_manager.cc:1367] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20240502 14:15:13.089787 20023 raft_consensus.cc:348] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:13.089972 20023 raft_consensus.cc:374] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.090034 20023 raft_consensus.cc:729] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.090246 20023 consensus_queue.cc:260] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:13.095029 20023 ts_tablet_manager.cc:1398] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:15:13.098992 13046 catalog_manager.cc:5484] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:13.099664 20025 tablet_bootstrap.cc:492] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:13.100334 20025 ts_tablet_manager.cc:1367] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.207s user 0.038s sys 0.033s
I20240502 14:15:13.100544 20024 tablet_bootstrap.cc:492] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:13.100685 20024 ts_tablet_manager.cc:1367] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.198s user 0.045s sys 0.085s
I20240502 14:15:13.101748 20024 raft_consensus.cc:348] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:13.101982 20024 raft_consensus.cc:374] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.102056 20024 raft_consensus.cc:729] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.102294 20024 consensus_queue.cc:260] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:13.103055 20024 ts_tablet_manager.cc:1398] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:13.103519 20025 raft_consensus.cc:348] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:13.104779 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:13.105345 13427 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:15:13.106516 20025 raft_consensus.cc:374] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.106585 20025 raft_consensus.cc:729] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.106775 20025 consensus_queue.cc:260] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:13.109155 20025 ts_tablet_manager.cc:1398] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240502 14:15:13.126331 19792 raft_consensus.cc:482] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:13.126502 19792 raft_consensus.cc:504] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:13.127158 19792 leader_election.cc:288] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:13.127418 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c065054cdb456da06172650abf0fae" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:13.127640 16155 raft_consensus.cc:2454] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:13.128836 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c065054cdb456da06172650abf0fae" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:13.129057 13578 raft_consensus.cc:2454] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:13.129446 16220 leader_election.cc:302] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:13.129717 19792 raft_consensus.cc:2781] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:13.129786 19792 raft_consensus.cc:482] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:13.129868 19792 raft_consensus.cc:3037] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.132513 19792 raft_consensus.cc:504] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:13.133301 16155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c065054cdb456da06172650abf0fae" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:13.133483 16155 raft_consensus.cc:3037] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.135968 16155 raft_consensus.cc:2454] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:13.136322 16220 leader_election.cc:302] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:13.136598 19959 raft_consensus.cc:2781] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:13.136727 19959 raft_consensus.cc:686] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:13.136988 19959 consensus_queue.cc:237] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:13.138718 19792 leader_election.cc:288] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:13.149775 16421 raft_consensus.cc:1261] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.150719 19413 consensus_queue.cc:1035] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.153875 16000 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:13.156708 13297 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240502 14:15:13.158309 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61c065054cdb456da06172650abf0fae" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:13.158546 13578 raft_consensus.cc:3037] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.162530 13578 raft_consensus.cc:2454] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:13.172462 13187 raft_consensus.cc:1261] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.172941 19413 consensus_queue.cc:1035] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.204118 13040 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet 771a509653534bfb914bf3a92667f35d (table rebalancer_dynamic_table [id=0ebdb2763bc648089775988f29782a77]). Expected version 1 got 0
I20240502 14:15:13.208215 13040 catalog_manager.cc:5484] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:13.218605 20061 tablet.cc:1722] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:13.219079 20062 tablet.cc:1722] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:13.219738 20062 tablet.cc:1620] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:15:13.220309 20061 tablet.cc:1620] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: MemRowSet was empty: no flush needed.
I20240502 14:15:13.222491 13045 catalog_manager.cc:5576] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303) does not have the latest schema for tablet 771a509653534bfb914bf3a92667f35d (table rebalancer_dynamic_table [id=0ebdb2763bc648089775988f29782a77]). Expected version 1 got 0
I20240502 14:15:13.223628 20059 mvcc.cc:205] Tried to move back new op lower bound from 7023244546644340736 to 7023244545817047040. Current Snapshot: MvccSnapshot[applied={T|T < 7023244546644340736}]
I20240502 14:15:13.227655 13042 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet 771a509653534bfb914bf3a92667f35d (table rebalancer_dynamic_table [id=0ebdb2763bc648089775988f29782a77]). Expected version 1 got 0
I20240502 14:15:13.239424 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=0ebdb2763bc648089775988f29782a77] alter complete (version 1)
I20240502 14:15:13.257558 13042 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet 771a509653534bfb914bf3a92667f35d (table rebalancer_dynamic_table [id=0ebdb2763bc648089775988f29782a77]). Expected version 1 got 0
I20240502 14:15:13.286877 20064 tablet.cc:1722] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:13.287552 20064 tablet.cc:1620] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:13.293578 13042 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet 771a509653534bfb914bf3a92667f35d (table rebalancer_dynamic_table [id=0ebdb2763bc648089775988f29782a77]). Expected version 1 got 0
I20240502 14:15:13.345830 19791 raft_consensus.cc:482] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:13.345984 19791 raft_consensus.cc:504] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:13.346592 19791 leader_election.cc:288] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:13.348574 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07012e715ded463db5edbb6019ee69cf" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:13.348793 16288 raft_consensus.cc:2454] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:13.349714 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07012e715ded463db5edbb6019ee69cf" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:13.349942 16421 raft_consensus.cc:2454] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:13.350217 16090 leader_election.cc:302] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:13.350603 18885 raft_consensus.cc:2781] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:13.350739 18885 raft_consensus.cc:482] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:13.350822 18885 raft_consensus.cc:3037] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.354133 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:13.355173 18885 raft_consensus.cc:504] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:13.356036 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07012e715ded463db5edbb6019ee69cf" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:13.356207 16288 raft_consensus.cc:3037] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.357579 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07012e715ded463db5edbb6019ee69cf" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:13.357776 16421 raft_consensus.cc:3037] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.358731 16288 raft_consensus.cc:2454] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:13.359315 16090 leader_election.cc:302] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:13.359875 19791 raft_consensus.cc:2781] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:13.360206 16421 raft_consensus.cc:2454] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:13.361397 18885 leader_election.cc:288] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:13.361925 19791 raft_consensus.cc:686] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:13.362170 19791 consensus_queue.cc:237] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:13.364926 20061 tablet.cc:1722] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:13.365341 20061 tablet.cc:1620] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: MemRowSet was empty: no flush needed.
I20240502 14:15:13.365458 20064 tablet.cc:1722] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:13.365818 20064 tablet.cc:1620] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:13.365932 20062 tablet.cc:1722] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:13.366465 20062 tablet.cc:1620] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:15:13.382423 13044 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=0ebdb2763bc648089775988f29782a77] alter complete (version 2)
I20240502 14:15:13.403622 13040 catalog_manager.cc:5484] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:13.431823 16020 consensus_queue.cc:237] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:13.433749 16421 raft_consensus.cc:1261] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:13.433945 15955 consensus_peers.cc:485] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2363a23cdde54893ae807ef3960dd032. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:13.434195 16288 raft_consensus.cc:1261] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.434401 19171 consensus_queue.cc:1035] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:13.434711 19904 consensus_queue.cc:1035] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:13.437289 19171 raft_consensus.cc:2932] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:13.441474 16288 raft_consensus.cc:2932] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:13.450568 13040 catalog_manager.cc:5484] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:13.456351 16421 raft_consensus.cc:2932] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:13.469311 16020 consensus_queue.cc:237] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:13.471310 16421 raft_consensus.cc:1261] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.471592 16155 raft_consensus.cc:1261] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.472098 19904 consensus_queue.cc:1035] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:13.472936 20074 consensus_queue.cc:1035] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:13.473819 15955 consensus_peers.cc:485] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 154fe82369ef4d2bb38e8f14637463bb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:13.476507 19171 raft_consensus.cc:2932] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:13.477361 16421 raft_consensus.cc:2932] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:13.477998 16155 raft_consensus.cc:2932] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:13.483364 13040 catalog_manager.cc:5484] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:13.484248 19171 consensus_queue.cc:1035] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.485874 13045 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:13.492731 20061 tablet.cc:1722] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:13.493794 20064 tablet.cc:1722] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:13.497838 20062 tablet.cc:1722] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:13.509891 13042 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=0ebdb2763bc648089775988f29782a77] alter complete (version 3)
I20240502 14:15:13.534453 19171 consensus_queue.cc:1035] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.544863 16421 consensus_queue.cc:237] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:13.546756 16288 raft_consensus.cc:1261] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:13.546900 16353 consensus_peers.cc:485] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 -> Peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Couldn't send request to peer 7eb4ecb2379b44c0954f2658c4a04350. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c28e61290d744d4c9409947f9e41706e. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:13.547206 16020 raft_consensus.cc:1261] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.547260 20097 consensus_queue.cc:1035] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:13.547732 19951 consensus_queue.cc:1035] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:13.549795 20097 raft_consensus.cc:2932] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:13.550415 16288 raft_consensus.cc:2932] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:13.551095 16020 raft_consensus.cc:2932] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:13.557737 13040 catalog_manager.cc:5484] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:13.579320 16288 consensus_queue.cc:237] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
W20240502 14:15:13.581562 16221 consensus_peers.cc:485] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2cb770f077c84cb99985ef59253f133b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:13.581661 16020 raft_consensus.cc:1261] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.581828 16421 raft_consensus.cc:1261] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.582171 20056 consensus_queue.cc:1035] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:13.582489 19792 consensus_queue.cc:1035] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:13.585012 19959 raft_consensus.cc:2932] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:13.585602 16421 raft_consensus.cc:2932] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:13.588567 16020 raft_consensus.cc:2932] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:13.593436 13040 catalog_manager.cc:5484] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:13.618934 16421 consensus_queue.cc:237] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:13.620801 13042 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:13.621013 13042 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
W20240502 14:15:13.621687 16353 consensus_peers.cc:485] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1c0b2fdd1d64bac8100d797c86fd8b2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:13.621901 16155 raft_consensus.cc:1261] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.622381 20100 consensus_queue.cc:1035] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:13.624114 16288 raft_consensus.cc:1261] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.624645 20100 consensus_queue.cc:1035] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:13.625681 13042 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 771a509653534bfb914bf3a92667f35d
I20240502 14:15:13.625679 20100 raft_consensus.cc:2932] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:13.626366 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 771a509653534bfb914bf3a92667f35d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:13 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:15:13.626971 20121 tablet_replica.cc:330] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:15:13.627104 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 771a509653534bfb914bf3a92667f35d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:13 UTC) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:13.627213 20121 raft_consensus.cc:2229] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:13.627455 20121 raft_consensus.cc:2258] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:13.628413 20121 ts_tablet_manager.cc:1875] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:13.631536 19792 consensus_queue.cc:1035] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.633324 16399 tablet_service.cc:1514] Processing DeleteTablet for tablet 771a509653534bfb914bf3a92667f35d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:13 UTC) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:13.633821 16155 raft_consensus.cc:2932] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:13.636364 20124 tablet_replica.cc:330] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:13.636593 20124 raft_consensus.cc:2229] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:13.636832 20124 raft_consensus.cc:2258] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:13.637725 20121 ts_tablet_manager.cc:1888] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:13.637921 20121 log.cc:1196] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/771a509653534bfb914bf3a92667f35d
I20240502 14:15:13.638386 20122 tablet_replica.cc:330] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:13.638440 20121 ts_tablet_manager.cc:1909] T 771a509653534bfb914bf3a92667f35d P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:15:13.638656 20122 raft_consensus.cc:2229] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:13.639086 20122 raft_consensus.cc:2258] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:13.639721 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 771a509653534bfb914bf3a92667f35d (table rebalancer_dynamic_table_new_name [id=0ebdb2763bc648089775988f29782a77]) successfully deleted
I20240502 14:15:13.643957 13042 catalog_manager.cc:5484] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:13.644995 20124 ts_tablet_manager.cc:1875] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:13.650583 20122 ts_tablet_manager.cc:1875] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:13.655033 19792 consensus_queue.cc:1035] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.660166 20122 ts_tablet_manager.cc:1888] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:13.660274 20122 log.cc:1196] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/771a509653534bfb914bf3a92667f35d
I20240502 14:15:13.660691 20122 ts_tablet_manager.cc:1909] T 771a509653534bfb914bf3a92667f35d P 88b77f6edc1c4a72b548bf47c05704f3: Deleting consensus metadata
I20240502 14:15:13.665863 20124 ts_tablet_manager.cc:1888] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:13.665978 20124 log.cc:1196] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/771a509653534bfb914bf3a92667f35d
I20240502 14:15:13.666471 20124 ts_tablet_manager.cc:1909] T 771a509653534bfb914bf3a92667f35d P 8b47d4390f8343a493d6c00123785968: Deleting consensus metadata
I20240502 14:15:13.667603 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 771a509653534bfb914bf3a92667f35d (table rebalancer_dynamic_table_new_name [id=0ebdb2763bc648089775988f29782a77]) successfully deleted
I20240502 14:15:13.671686 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 771a509653534bfb914bf3a92667f35d (table rebalancer_dynamic_table_new_name [id=0ebdb2763bc648089775988f29782a77]) successfully deleted
I20240502 14:15:13.675552 16287 raft_consensus.cc:2932] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:13.700896 16421 consensus_queue.cc:237] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:13.703049 16020 raft_consensus.cc:1261] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:15:13.703543 16152 raft_consensus.cc:1261] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:15:13.703603 20101 consensus_queue.cc:1035] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:13.704092 16353 consensus_peers.cc:485] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 -> Peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Couldn't send request to peer 7eb4ecb2379b44c0954f2658c4a04350. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7a740ff5e0964efcb1fd7dd54dc4094c. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:13.704308 20101 consensus_queue.cc:1035] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:13.710529 20101 raft_consensus.cc:2932] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:13.712015 16023 raft_consensus.cc:2932] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:13.712999 16152 raft_consensus.cc:2932] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:13.720031 13042 catalog_manager.cc:5484] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New cstate: current_term: 2 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:13.733843 13042 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:13.739918 13578 consensus_queue.cc:237] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:13.741290 16265 tablet_service.cc:1467] Processing CreateTablet for tablet d587d24c7cc04e268d6124aafd8ff04a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72f5b283fad34ca3b10e5016af6482cb]), partition=
I20240502 14:15:13.741678 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d587d24c7cc04e268d6124aafd8ff04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.741683 13559 tablet_service.cc:1467] Processing CreateTablet for tablet d587d24c7cc04e268d6124aafd8ff04a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72f5b283fad34ca3b10e5016af6482cb]), partition=
I20240502 14:15:13.741948 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d587d24c7cc04e268d6124aafd8ff04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.748255 13297 tablet_service.cc:1467] Processing CreateTablet for tablet d587d24c7cc04e268d6124aafd8ff04a (DEFAULT_TABLE table=rebalancer_dynamic_table [id=72f5b283fad34ca3b10e5016af6482cb]), partition=
I20240502 14:15:13.748647 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d587d24c7cc04e268d6124aafd8ff04a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.750924 20143 tablet_bootstrap.cc:492] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:13.752166 20143 tablet_bootstrap.cc:654] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.753939 20143 tablet_bootstrap.cc:492] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:13.754097 20143 ts_tablet_manager.cc:1367] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:13.754774 20143 raft_consensus.cc:348] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:13.754956 20143 raft_consensus.cc:374] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.755057 20143 raft_consensus.cc:729] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.755265 20143 consensus_queue.cc:260] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:13.755769 20143 ts_tablet_manager.cc:1398] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:13.756723 16152 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:15:13.756901 13516 consensus_peers.cc:485] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16c817b864c4425eaa85ac4b5aff59fa. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:13.757025 13188 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:13.757187 20030 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:13.757462 19613 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:13.758463 20146 tablet_bootstrap.cc:492] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:13.759898 20146 tablet_bootstrap.cc:654] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.765890 20145 tablet_bootstrap.cc:492] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:13.768056 20145 tablet_bootstrap.cc:654] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.770416 19613 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:13.770963 16152 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:13.778683 20145 tablet_bootstrap.cc:492] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:13.778843 20145 ts_tablet_manager.cc:1367] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.003s
I20240502 14:15:13.779099 13040 catalog_manager.cc:5484] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:13.779959 20145 raft_consensus.cc:348] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:13.780216 20145 raft_consensus.cc:374] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.780319 20145 raft_consensus.cc:729] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.780647 20145 consensus_queue.cc:260] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:13.781280 20145 ts_tablet_manager.cc:1398] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:13.794867 20146 tablet_bootstrap.cc:492] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:13.795038 20146 ts_tablet_manager.cc:1367] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.014s
I20240502 14:15:13.794780 13188 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:13.795795 20146 raft_consensus.cc:348] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:13.796034 20146 raft_consensus.cc:374] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.796119 20146 raft_consensus.cc:729] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.796357 20146 consensus_queue.cc:260] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:13.796941 20146 ts_tablet_manager.cc:1398] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:13.811599 13578 consensus_queue.cc:237] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:13.814385 19791 consensus_queue.cc:1035] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.821521 18885 consensus_queue.cc:1035] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:13.826027 20030 raft_consensus.cc:482] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:13.826153 20030 raft_consensus.cc:504] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:13.826833 20030 leader_election.cc:288] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:13.827433 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d587d24c7cc04e268d6124aafd8ff04a" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:13.827466 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d587d24c7cc04e268d6124aafd8ff04a" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:13.827636 16287 raft_consensus.cc:2454] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:13.827687 13319 raft_consensus.cc:2454] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:13.828190 13516 leader_election.cc:302] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:15:13.828714 20030 raft_consensus.cc:2781] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:13.828789 20030 raft_consensus.cc:482] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:13.828850 20030 raft_consensus.cc:3037] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.831745 20030 raft_consensus.cc:504] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:13.832720 16287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d587d24c7cc04e268d6124aafd8ff04a" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:13.832949 16287 raft_consensus.cc:3037] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.835533 13447 raft_consensus.cc:1261] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.836037 19613 consensus_queue.cc:1035] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:13.838274 16287 raft_consensus.cc:2454] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:13.838321 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d587d24c7cc04e268d6124aafd8ff04a" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:13.838521 13319 raft_consensus.cc:3037] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:13.841506 13516 leader_election.cc:302] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:15:13.842672 16023 raft_consensus.cc:1261] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:13.843565 19613 raft_consensus.cc:2781] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
W20240502 14:15:13.844223 13516 consensus_peers.cc:485] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cee69dfc4ded45de81cfd8b06c5f2f2f. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:13.847133 20160 consensus_queue.cc:1035] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:13.841176 13319 raft_consensus.cc:2454] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:13.847621 13447 raft_consensus.cc:2932] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:13.849690 20030 leader_election.cc:288] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:13.849436 20163 raft_consensus.cc:2932] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:13.855497 13042 catalog_manager.cc:5484] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:13.874349 19613 raft_consensus.cc:686] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:15:13.874645 19613 consensus_queue.cc:237] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:13.874960 16023 raft_consensus.cc:2932] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:13.886731 16421 consensus_queue.cc:237] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:13.888932 13319 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:15:13.889626 16353 consensus_peers.cc:485] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 -> Peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Couldn't send request to peer 7eb4ecb2379b44c0954f2658c4a04350. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b34f2a22306459784a1ad7b81cd1683. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:13.889887 20100 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:13.891095 16287 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:13.891503 20101 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:13.892890 19951 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:13.893322 13319 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:13.902866 13042 catalog_manager.cc:5484] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 4 to 5, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:13.919036 16288 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:13.931838 13040 catalog_manager.cc:5484] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:13.932788 13042 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:13.945852 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 4e05e25ef22e40d380ab7f83437e1f4b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c3c20e34fcc944289e4ea927676952cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.946256 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e05e25ef22e40d380ab7f83437e1f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.946300 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3b25a9e8b049a88bb999d9362e98da (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c3c20e34fcc944289e4ea927676952cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.946751 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3b25a9e8b049a88bb999d9362e98da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.949600 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 4e05e25ef22e40d380ab7f83437e1f4b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c3c20e34fcc944289e4ea927676952cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.949992 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e05e25ef22e40d380ab7f83437e1f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.954751 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3b25a9e8b049a88bb999d9362e98da (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c3c20e34fcc944289e4ea927676952cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.955166 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3b25a9e8b049a88bb999d9362e98da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.959837 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3b25a9e8b049a88bb999d9362e98da (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c3c20e34fcc944289e4ea927676952cf]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.960381 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3b25a9e8b049a88bb999d9362e98da. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.964892 13429 tablet_service.cc:1467] Processing CreateTablet for tablet e93e561ce7ec4e6fa7e211fd1e4fba56 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c3c20e34fcc944289e4ea927676952cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.965396 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 4e05e25ef22e40d380ab7f83437e1f4b (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c3c20e34fcc944289e4ea927676952cf]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.965210 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93e561ce7ec4e6fa7e211fd1e4fba56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.965775 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e05e25ef22e40d380ab7f83437e1f4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.966638 16399 tablet_service.cc:1467] Processing CreateTablet for tablet e93e561ce7ec4e6fa7e211fd1e4fba56 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c3c20e34fcc944289e4ea927676952cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.966997 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93e561ce7ec4e6fa7e211fd1e4fba56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.969826 16000 tablet_service.cc:1467] Processing CreateTablet for tablet e93e561ce7ec4e6fa7e211fd1e4fba56 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=c3c20e34fcc944289e4ea927676952cf]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:13.970206 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e93e561ce7ec4e6fa7e211fd1e4fba56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:13.973146 20145 tablet_bootstrap.cc:492] T 4e05e25ef22e40d380ab7f83437e1f4b P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:13.974342 20145 tablet_bootstrap.cc:654] T 4e05e25ef22e40d380ab7f83437e1f4b P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.984946 20146 tablet_bootstrap.cc:492] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:13.986867 20146 tablet_bootstrap.cc:654] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.989276 20145 tablet_bootstrap.cc:492] T 4e05e25ef22e40d380ab7f83437e1f4b P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:13.989440 20145 ts_tablet_manager.cc:1367] T 4e05e25ef22e40d380ab7f83437e1f4b P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.003s
I20240502 14:15:13.990794 20145 raft_consensus.cc:348] T 4e05e25ef22e40d380ab7f83437e1f4b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:13.991086 20145 raft_consensus.cc:374] T 4e05e25ef22e40d380ab7f83437e1f4b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.991189 20145 raft_consensus.cc:729] T 4e05e25ef22e40d380ab7f83437e1f4b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.991468 20145 consensus_queue.cc:260] T 4e05e25ef22e40d380ab7f83437e1f4b P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:13.992100 20146 tablet_bootstrap.cc:492] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:13.992214 20146 ts_tablet_manager.cc:1367] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.001s
I20240502 14:15:13.992825 20185 tablet_bootstrap.cc:492] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:13.993074 20146 raft_consensus.cc:348] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:13.993235 20146 raft_consensus.cc:374] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:13.993281 20146 raft_consensus.cc:729] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:13.993448 20146 consensus_queue.cc:260] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:13.994050 20146 ts_tablet_manager.cc:1398] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:13.994911 20182 tablet_bootstrap.cc:492] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:13.996292 20182 tablet_bootstrap.cc:654] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.996912 20185 tablet_bootstrap.cc:654] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:13.997663 20180 tablet_bootstrap.cc:492] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:13.997887 20145 ts_tablet_manager.cc:1398] T 4e05e25ef22e40d380ab7f83437e1f4b P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:15:13.999383 20180 tablet_bootstrap.cc:654] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.001804 20184 tablet_bootstrap.cc:492] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:14.005400 20184 tablet_bootstrap.cc:654] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.005590 20180 tablet_bootstrap.cc:492] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:14.008608 19613 raft_consensus.cc:482] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:14.008718 19613 raft_consensus.cc:504] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:14.009284 19613 leader_election.cc:288] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:14.009735 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e05e25ef22e40d380ab7f83437e1f4b" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:14.009941 13319 raft_consensus.cc:2454] T 4e05e25ef22e40d380ab7f83437e1f4b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:14.010378 13516 leader_election.cc:302] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:14.010680 19613 raft_consensus.cc:2781] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:14.010766 19613 raft_consensus.cc:482] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:14.010871 19613 raft_consensus.cc:3037] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.012915 20186 tablet_bootstrap.cc:492] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:14.013257 19613 raft_consensus.cc:504] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:14.013790 19613 leader_election.cc:288] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:14.014214 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e05e25ef22e40d380ab7f83437e1f4b" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:14.014369 13319 raft_consensus.cc:3037] T 4e05e25ef22e40d380ab7f83437e1f4b P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.017971 13319 raft_consensus.cc:2454] T 4e05e25ef22e40d380ab7f83437e1f4b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:14.018386 13516 leader_election.cc:302] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:14.018828 20184 tablet_bootstrap.cc:492] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:14.019016 20162 raft_consensus.cc:2781] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:14.019147 20162 raft_consensus.cc:686] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:15:14.019449 20162 consensus_queue.cc:237] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:14.020298 20184 ts_tablet_manager.cc:1367] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.009s
I20240502 14:15:14.022361 20184 raft_consensus.cc:348] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:14.022544 20186 tablet_bootstrap.cc:654] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.023849 20184 raft_consensus.cc:374] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.025502 20184 raft_consensus.cc:729] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.025584 13040 catalog_manager.cc:5484] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:14.025861 20184 consensus_queue.cc:260] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:14.039948 20184 ts_tablet_manager.cc:1398] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 14:15:14.040323 20195 ts_tablet_manager.cc:902] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3: Initiating tablet copy from peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:14.045061 20186 tablet_bootstrap.cc:492] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:14.045317 20186 ts_tablet_manager.cc:1367] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240502 14:15:14.046134 20186 raft_consensus.cc:348] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:14.046356 20182 tablet_bootstrap.cc:492] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:14.046391 20186 raft_consensus.cc:374] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.046511 20186 raft_consensus.cc:729] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.046542 20182 ts_tablet_manager.cc:1367] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.000s
I20240502 14:15:14.046764 20186 consensus_queue.cc:260] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:14.047384 20186 ts_tablet_manager.cc:1398] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:14.047322 20182 raft_consensus.cc:348] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:14.047524 20182 raft_consensus.cc:374] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.047581 20182 raft_consensus.cc:729] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.047777 20182 consensus_queue.cc:260] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:14.048285 20182 ts_tablet_manager.cc:1398] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:14.050143 20182 tablet_bootstrap.cc:492] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:14.051508 20182 tablet_bootstrap.cc:654] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.062093 20190 ts_tablet_manager.cc:902] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3: Initiating tablet copy from peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:14.009943 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e05e25ef22e40d380ab7f83437e1f4b" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:14.016607 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e05e25ef22e40d380ab7f83437e1f4b" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
W20240502 14:15:14.064755 13514 leader_election.cc:341] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:15:14.065115 13514 leader_election.cc:341] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:14.062873 20192 ts_tablet_manager.cc:902] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:14.066095 20180 ts_tablet_manager.cc:1367] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.000s
I20240502 14:15:14.066890 20180 raft_consensus.cc:348] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:14.067162 20180 raft_consensus.cc:374] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.067307 20180 raft_consensus.cc:729] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.067656 20180 consensus_queue.cc:260] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:14.070605 20182 tablet_bootstrap.cc:492] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:14.070750 20182 ts_tablet_manager.cc:1367] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.014s
I20240502 14:15:14.071480 13042 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:14.071611 20182 raft_consensus.cc:348] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:14.071864 20182 raft_consensus.cc:374] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.071918 20182 raft_consensus.cc:729] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.072168 20182 consensus_queue.cc:260] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:14.074187 20182 ts_tablet_manager.cc:1398] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:15:14.077054 13319 raft_consensus.cc:1261] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:14.077412 16288 raft_consensus.cc:1261] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:14.077579 19613 consensus_queue.cc:1035] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.077922 20162 consensus_queue.cc:1035] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.089154 13040 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet d587d24c7cc04e268d6124aafd8ff04a (table rebalancer_dynamic_table [id=72f5b283fad34ca3b10e5016af6482cb]). Expected version 1 got 0
I20240502 14:15:14.096562 20199 ts_tablet_manager.cc:902] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350: Initiating tablet copy from peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:14.097033 20180 ts_tablet_manager.cc:1398] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.031s user 0.002s sys 0.000s
I20240502 14:15:14.098873 20180 tablet_bootstrap.cc:492] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:14.100514 20180 tablet_bootstrap.cc:654] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.112447 20185 tablet_bootstrap.cc:492] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:14.110879 20204 tablet.cc:1722] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:14.114137 16023 consensus_queue.cc:237] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:14.122649 20185 ts_tablet_manager.cc:1367] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.130s user 0.016s sys 0.028s
I20240502 14:15:14.123765 20185 raft_consensus.cc:348] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:14.125972 20185 raft_consensus.cc:374] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.129112 13040 catalog_manager.cc:5576] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817) does not have the latest schema for tablet d587d24c7cc04e268d6124aafd8ff04a (table rebalancer_dynamic_table [id=72f5b283fad34ca3b10e5016af6482cb]). Expected version 1 got 0
I20240502 14:15:14.134891 20190 tablet_copy_client.cc:308] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.197:43695
W20240502 14:15:14.134945 15956 consensus_peers.cc:485] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb472259eca84c31b2ff9f33f7e61177. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:14.135217 20185 raft_consensus.cc:729] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.135620 16285 raft_consensus.cc:1261] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:14.136020 16043 tablet_copy_service.cc:140] P d1da7f11f0604f78934354a992bb0f55: Received BeginTabletCopySession request for tablet 2363a23cdde54893ae807ef3960dd032 from peer 88b77f6edc1c4a72b548bf47c05704f3 ({username='slave'} at 127.6.196.195:47189)
I20240502 14:15:14.136236 19904 consensus_queue.cc:1035] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:14.136240 16043 tablet_copy_service.cc:161] P d1da7f11f0604f78934354a992bb0f55: Beginning new tablet copy session on tablet 2363a23cdde54893ae807ef3960dd032 from peer 88b77f6edc1c4a72b548bf47c05704f3 at {username='slave'} at 127.6.196.195:47189: session id = 88b77f6edc1c4a72b548bf47c05704f3-2363a23cdde54893ae807ef3960dd032
I20240502 14:15:14.136523 16421 raft_consensus.cc:1261] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:14.137243 20185 consensus_queue.cc:260] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:14.137331 19904 consensus_queue.cc:1035] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:14.139266 16043 tablet_copy_source_session.cc:205] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:14.141463 20190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2363a23cdde54893ae807ef3960dd032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.142057 20185 ts_tablet_manager.cc:1398] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240502 14:15:14.142277 20204 tablet.cc:1620] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: MemRowSet was empty: no flush needed.
I20240502 14:15:14.146337 19904 raft_consensus.cc:2932] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:14.148665 20180 tablet_bootstrap.cc:492] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:14.148877 20180 ts_tablet_manager.cc:1367] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20240502 14:15:14.149236 20192 tablet_copy_client.cc:308] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.197:43695
I20240502 14:15:14.149806 16043 tablet_copy_service.cc:140] P d1da7f11f0604f78934354a992bb0f55: Received BeginTabletCopySession request for tablet 154fe82369ef4d2bb38e8f14637463bb from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:54191)
I20240502 14:15:14.150014 16043 tablet_copy_service.cc:161] P d1da7f11f0604f78934354a992bb0f55: Beginning new tablet copy session on tablet 154fe82369ef4d2bb38e8f14637463bb from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:54191: session id = 20a967bad5504b1e9c6fb720ecdd1625-154fe82369ef4d2bb38e8f14637463bb
I20240502 14:15:14.149755 16421 raft_consensus.cc:2932] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:14.151846 20190 tablet_copy_client.cc:783] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:14.152034 20190 tablet_copy_client.cc:647] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:14.153478 20190 tablet_copy_client.cc:523] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:14.153525 16043 tablet_copy_source_session.cc:205] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:14.154685 20192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 154fe82369ef4d2bb38e8f14637463bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.156332 20190 tablet_bootstrap.cc:492] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:14.161075 13045 catalog_manager.cc:5484] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:14.162778 20192 tablet_copy_client.cc:783] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:14.163033 20192 tablet_copy_client.cc:647] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:14.163251 20180 raft_consensus.cc:348] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:14.163519 20180 raft_consensus.cc:374] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.163637 20180 raft_consensus.cc:729] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.163920 20180 consensus_queue.cc:260] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:14.167285 20180 ts_tablet_manager.cc:1398] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.018s user 0.000s sys 0.001s
I20240502 14:15:14.173780 13046 catalog_manager.cc:5576] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193) does not have the latest schema for tablet d587d24c7cc04e268d6124aafd8ff04a (table rebalancer_dynamic_table [id=72f5b283fad34ca3b10e5016af6482cb]). Expected version 1 got 0
I20240502 14:15:14.175276 20192 tablet_copy_client.cc:523] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:14.185964 13045 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=72f5b283fad34ca3b10e5016af6482cb] alter complete (version 1)
I20240502 14:15:14.186987 20192 tablet_bootstrap.cc:492] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:14.188387 20207 ts_tablet_manager.cc:902] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350: Initiating tablet copy from peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:14.193037 19951 raft_consensus.cc:482] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:14.193203 19951 raft_consensus.cc:504] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:14.194016 19951 leader_election.cc:288] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:14.195466 16285 raft_consensus.cc:2932] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:14.197540 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e561ce7ec4e6fa7e211fd1e4fba56" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:14.197743 13447 raft_consensus.cc:2454] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:15:14.198235 16353 leader_election.cc:302] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:14.200028 20101 raft_consensus.cc:2781] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:14.200135 20101 raft_consensus.cc:482] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:14.200253 20101 raft_consensus.cc:3037] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.200906 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e561ce7ec4e6fa7e211fd1e4fba56" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:14.201104 16023 raft_consensus.cc:2454] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 0.
I20240502 14:15:14.202443 20205 tablet.cc:1722] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:14.202865 20101 raft_consensus.cc:504] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:14.203956 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e561ce7ec4e6fa7e211fd1e4fba56" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:14.204164 16023 raft_consensus.cc:3037] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.205960 20205 tablet.cc:1620] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:14.209837 20207 tablet_copy_client.cc:308] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.200:43857
I20240502 14:15:14.210114 13045 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet d587d24c7cc04e268d6124aafd8ff04a (table rebalancer_dynamic_table [id=72f5b283fad34ca3b10e5016af6482cb]). Expected version 1 got 0
I20240502 14:15:14.211472 20195 tablet_copy_client.cc:308] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.200:43857
I20240502 14:15:14.212075 16442 tablet_copy_service.cc:140] P 8b47d4390f8343a493d6c00123785968: Received BeginTabletCopySession request for tablet 7a740ff5e0964efcb1fd7dd54dc4094c from peer 7eb4ecb2379b44c0954f2658c4a04350 ({username='slave'} at 127.6.196.196:52755)
I20240502 14:15:14.212213 16442 tablet_copy_service.cc:161] P 8b47d4390f8343a493d6c00123785968: Beginning new tablet copy session on tablet 7a740ff5e0964efcb1fd7dd54dc4094c from peer 7eb4ecb2379b44c0954f2658c4a04350 at {username='slave'} at 127.6.196.196:52755: session id = 7eb4ecb2379b44c0954f2658c4a04350-7a740ff5e0964efcb1fd7dd54dc4094c
I20240502 14:15:14.212361 20226 ts_tablet_manager.cc:902] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d: Initiating tablet copy from peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:14.215023 16442 tablet_copy_source_session.cc:205] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:14.218183 20224 tablet.cc:1722] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:14.219630 16023 raft_consensus.cc:2454] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:15:14.220078 16355 leader_election.cc:302] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:14.220355 16442 tablet_copy_service.cc:140] P 8b47d4390f8343a493d6c00123785968: Received BeginTabletCopySession request for tablet a1c0b2fdd1d64bac8100d797c86fd8b2 from peer 88b77f6edc1c4a72b548bf47c05704f3 ({username='slave'} at 127.6.196.195:39073)
I20240502 14:15:14.220407 19951 raft_consensus.cc:2781] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:14.220513 16442 tablet_copy_service.cc:161] P 8b47d4390f8343a493d6c00123785968: Beginning new tablet copy session on tablet a1c0b2fdd1d64bac8100d797c86fd8b2 from peer 88b77f6edc1c4a72b548bf47c05704f3 at {username='slave'} at 127.6.196.195:39073: session id = 88b77f6edc1c4a72b548bf47c05704f3-a1c0b2fdd1d64bac8100d797c86fd8b2
I20240502 14:15:14.220577 19951 raft_consensus.cc:686] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:15:14.220827 19951 consensus_queue.cc:237] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:14.224294 20101 leader_election.cc:288] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:14.224418 16442 tablet_copy_source_session.cc:205] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:14.224960 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e561ce7ec4e6fa7e211fd1e4fba56" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:14.225299 13447 raft_consensus.cc:3037] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.225615 20207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a740ff5e0964efcb1fd7dd54dc4094c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.228623 13447 raft_consensus.cc:2454] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 1.
I20240502 14:15:14.232435 20195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1c0b2fdd1d64bac8100d797c86fd8b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.235874 20224 tablet.cc:1620] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: MemRowSet was empty: no flush needed.
I20240502 14:15:14.236968 13045 catalog_manager.cc:5576] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193) does not have the latest schema for tablet d587d24c7cc04e268d6124aafd8ff04a (table rebalancer_dynamic_table [id=72f5b283fad34ca3b10e5016af6482cb]). Expected version 1 got 0
I20240502 14:15:14.241526 20192 tablet_bootstrap.cc:492] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:14.241941 20192 tablet_bootstrap.cc:492] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:15:14.242179 20192 ts_tablet_manager.cc:1367] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.000s
I20240502 14:15:14.245728 20199 tablet_copy_client.cc:308] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.200:43857
I20240502 14:15:14.253055 16442 tablet_copy_service.cc:140] P 8b47d4390f8343a493d6c00123785968: Received BeginTabletCopySession request for tablet c28e61290d744d4c9409947f9e41706e from peer 7eb4ecb2379b44c0954f2658c4a04350 ({username='slave'} at 127.6.196.196:52755)
I20240502 14:15:14.253288 16442 tablet_copy_service.cc:161] P 8b47d4390f8343a493d6c00123785968: Beginning new tablet copy session on tablet c28e61290d744d4c9409947f9e41706e from peer 7eb4ecb2379b44c0954f2658c4a04350 at {username='slave'} at 127.6.196.196:52755: session id = 7eb4ecb2379b44c0954f2658c4a04350-c28e61290d744d4c9409947f9e41706e
I20240502 14:15:14.252411 13045 catalog_manager.cc:5484] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:14.256099 16442 tablet_copy_source_session.cc:205] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:14.257752 20199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c28e61290d744d4c9409947f9e41706e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.257282 20195 tablet_copy_client.cc:783] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:14.258864 20195 tablet_copy_client.cc:647] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:14.265473 20199 tablet_copy_client.cc:783] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:14.266140 20199 tablet_copy_client.cc:647] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:14.267894 20195 tablet_copy_client.cc:523] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:14.271111 20199 tablet_copy_client.cc:523] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:14.271539 20195 tablet_bootstrap.cc:492] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:14.277737 20199 tablet_bootstrap.cc:492] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:14.283648 20226 tablet_copy_client.cc:308] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.199:33857
I20240502 14:15:14.284397 16308 tablet_copy_service.cc:140] P 5297e816f782430d9a01220f0471a882: Received BeginTabletCopySession request for tablet 2cb770f077c84cb99985ef59253f133b from peer cd67493f6f0142f896010f2b8a39887d ({username='slave'} at 127.6.196.194:56211)
I20240502 14:15:14.284595 16308 tablet_copy_service.cc:161] P 5297e816f782430d9a01220f0471a882: Beginning new tablet copy session on tablet 2cb770f077c84cb99985ef59253f133b from peer cd67493f6f0142f896010f2b8a39887d at {username='slave'} at 127.6.196.194:56211: session id = cd67493f6f0142f896010f2b8a39887d-2cb770f077c84cb99985ef59253f133b
I20240502 14:15:14.287533 16308 tablet_copy_source_session.cc:205] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:14.288930 20226 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cb770f077c84cb99985ef59253f133b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.294941 20226 tablet_copy_client.cc:783] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:14.295207 20226 tablet_copy_client.cc:647] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:14.297657 20065 raft_consensus.cc:482] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:14.297857 20065 raft_consensus.cc:504] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:14.298702 20065 leader_election.cc:288] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:14.299687 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3b25a9e8b049a88bb999d9362e98da" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:14.299731 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3b25a9e8b049a88bb999d9362e98da" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:14.299916 13447 raft_consensus.cc:2454] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:14.300033 13186 raft_consensus.cc:2454] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 0.
I20240502 14:15:14.300446 16088 leader_election.cc:302] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:14.301059 20065 raft_consensus.cc:2781] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:14.301172 20065 raft_consensus.cc:482] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:14.301280 20065 raft_consensus.cc:3037] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.302060 20190 tablet_bootstrap.cc:492] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:14.306739 13045 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:14.307145 20065 raft_consensus.cc:504] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:14.307655 20065 leader_election.cc:288] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:14.309536 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3b25a9e8b049a88bb999d9362e98da" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:14.242781 20192 raft_consensus.cc:348] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:14.309723 13186 raft_consensus.cc:3037] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.309800 20192 raft_consensus.cc:729] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:15:14.309789 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3b25a9e8b049a88bb999d9362e98da" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:14.309921 13447 raft_consensus.cc:3037] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.309991 20192 consensus_queue.cc:260] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:14.312175 13447 raft_consensus.cc:2454] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:14.312470 13186 raft_consensus.cc:2454] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86f7440570f4796babfd44c3d7fd847 in term 1.
I20240502 14:15:14.312492 16088 leader_election.cc:302] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:14.314421 18885 raft_consensus.cc:2781] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:14.314543 18885 raft_consensus.cc:686] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Becoming Leader. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Running, Role: LEADER
I20240502 14:15:14.314791 18885 consensus_queue.cc:237] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:14.315699 20192 ts_tablet_manager.cc:1398] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.073s user 0.000s sys 0.002s
I20240502 14:15:14.315747 20195 tablet_bootstrap.cc:492] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:14.322266 20204 tablet.cc:1722] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:14.322662 20204 tablet.cc:1620] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: MemRowSet was empty: no flush needed.
I20240502 14:15:14.327520 16043 tablet_copy_service.cc:342] P d1da7f11f0604f78934354a992bb0f55: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-154fe82369ef4d2bb38e8f14637463bb received from {username='slave'} at 127.6.196.193:54191
I20240502 14:15:14.327673 16043 tablet_copy_service.cc:433] P d1da7f11f0604f78934354a992bb0f55: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-154fe82369ef4d2bb38e8f14637463bb on tablet 154fe82369ef4d2bb38e8f14637463bb with peer 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:15:14.330317 20249 ts_tablet_manager.cc:902] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:14.330296 20224 tablet.cc:1722] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:14.330708 20224 tablet.cc:1620] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: MemRowSet was empty: no flush needed.
I20240502 14:15:14.332706 13040 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=72f5b283fad34ca3b10e5016af6482cb] alter complete (version 2)
I20240502 14:15:14.333199 20177 raft_consensus.cc:482] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:14.333335 20177 raft_consensus.cc:504] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:14.334059 20177 leader_election.cc:288] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:14.334466 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e05e25ef22e40d380ab7f83437e1f4b" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:14.335829 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e05e25ef22e40d380ab7f83437e1f4b" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:14.336007 13318 raft_consensus.cc:2379] T 4e05e25ef22e40d380ab7f83437e1f4b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 20a967bad5504b1e9c6fb720ecdd1625 in current term 1: Already voted for candidate 7eb4ecb2379b44c0954f2658c4a04350 in this term.
I20240502 14:15:14.336349 13122 leader_election.cc:302] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:14.336699 20177 raft_consensus.cc:3037] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:14.338966 20177 raft_consensus.cc:2726] T 4e05e25ef22e40d380ab7f83437e1f4b P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:14.340201 13045 catalog_manager.cc:5484] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86f7440570f4796babfd44c3d7fd847 (127.6.196.198). New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:14.347002 20190 tablet_bootstrap.cc:492] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap complete.
I20240502 14:15:14.347107 20195 tablet_bootstrap.cc:492] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap complete.
I20240502 14:15:14.347276 20190 ts_tablet_manager.cc:1367] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.191s user 0.004s sys 0.009s
I20240502 14:15:14.347307 20195 ts_tablet_manager.cc:1367] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.076s user 0.011s sys 0.002s
I20240502 14:15:14.347923 20190 raft_consensus.cc:348] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:14.348111 20190 raft_consensus.cc:729] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: LEARNER
I20240502 14:15:14.348059 20195 raft_consensus.cc:348] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:14.352725 20195 raft_consensus.cc:729] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: LEARNER
I20240502 14:15:14.352901 20195 consensus_queue.cc:260] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:14.354316 20190 consensus_queue.cc:260] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:14.355569 20190 ts_tablet_manager.cc:1398] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:15:14.356010 20249 tablet_copy_client.cc:308] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:15:14.356217 16043 tablet_copy_service.cc:342] P d1da7f11f0604f78934354a992bb0f55: Request end of tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-2363a23cdde54893ae807ef3960dd032 received from {username='slave'} at 127.6.196.195:47189
I20240502 14:15:14.356313 16043 tablet_copy_service.cc:433] P d1da7f11f0604f78934354a992bb0f55: ending tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-2363a23cdde54893ae807ef3960dd032 on tablet 2363a23cdde54893ae807ef3960dd032 with peer 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:15:14.358083 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet 16c817b864c4425eaa85ac4b5aff59fa from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:40701)
I20240502 14:15:14.358211 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet 16c817b864c4425eaa85ac4b5aff59fa from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:40701: session id = d1da7f11f0604f78934354a992bb0f55-16c817b864c4425eaa85ac4b5aff59fa
I20240502 14:15:14.360930 13601 tablet_copy_source_session.cc:205] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:14.361553 20195 ts_tablet_manager.cc:1398] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240502 14:15:14.365567 20249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c817b864c4425eaa85ac4b5aff59fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.368707 20207 tablet_copy_client.cc:783] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:14.368921 20207 tablet_copy_client.cc:647] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:14.370306 20207 tablet_copy_client.cc:523] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:14.377301 20207 tablet_bootstrap.cc:492] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:14.379185 16442 tablet_copy_service.cc:342] P 8b47d4390f8343a493d6c00123785968: Request end of tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-a1c0b2fdd1d64bac8100d797c86fd8b2 received from {username='slave'} at 127.6.196.195:39073
I20240502 14:15:14.379309 16442 tablet_copy_service.cc:433] P 8b47d4390f8343a493d6c00123785968: ending tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-a1c0b2fdd1d64bac8100d797c86fd8b2 on tablet a1c0b2fdd1d64bac8100d797c86fd8b2 with peer 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:15:14.386241 20248 ts_tablet_manager.cc:902] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350: Initiating tablet copy from peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:14.388777 20247 ts_tablet_manager.cc:902] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:14.393711 20199 tablet_bootstrap.cc:492] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:14.394119 20199 tablet_bootstrap.cc:492] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap complete.
I20240502 14:15:14.394387 20199 ts_tablet_manager.cc:1367] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.117s user 0.004s sys 0.020s
I20240502 14:15:14.395162 20199 raft_consensus.cc:348] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:14.395416 20199 raft_consensus.cc:729] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: LEARNER
I20240502 14:15:14.395602 20199 consensus_queue.cc:260] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:14.398654 20199 ts_tablet_manager.cc:1398] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240502 14:15:14.399407 16442 tablet_copy_service.cc:342] P 8b47d4390f8343a493d6c00123785968: Request end of tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-c28e61290d744d4c9409947f9e41706e received from {username='slave'} at 127.6.196.196:52755
I20240502 14:15:14.399541 16442 tablet_copy_service.cc:433] P 8b47d4390f8343a493d6c00123785968: ending tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-c28e61290d744d4c9409947f9e41706e on tablet c28e61290d744d4c9409947f9e41706e with peer 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:15:14.405774 13447 raft_consensus.cc:1203] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:14.410300 20249 tablet_copy_client.cc:783] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:14.410576 20249 tablet_copy_client.cc:647] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:14.412616 20249 tablet_copy_client.cc:523] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:14.413770 20226 tablet_copy_client.cc:523] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:14.416846 20226 tablet_bootstrap.cc:492] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:14.435994 13045 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:14.437754 20205 tablet.cc:1722] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:14.438243 20205 tablet.cc:1620] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:14.441121 13580 raft_consensus.cc:1203] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:14.443931 20162 consensus_queue.cc:1035] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.456784 20249 tablet_bootstrap.cc:492] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:14.468739 20226 tablet_bootstrap.cc:492] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:14.469172 20226 tablet_bootstrap.cc:492] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d: Bootstrap complete.
I20240502 14:15:14.471000 20226 ts_tablet_manager.cc:1367] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.004s
I20240502 14:15:14.471503 20204 tablet.cc:1722] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:14.471696 20226 raft_consensus.cc:348] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:14.471884 20226 raft_consensus.cc:729] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: LEARNER
I20240502 14:15:14.472049 20226 consensus_queue.cc:260] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:14.475531 20226 ts_tablet_manager.cc:1398] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:15:14.478061 20224 tablet.cc:1722] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:14.482131 13046 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet d587d24c7cc04e268d6124aafd8ff04a (table rebalancer_dynamic_table_new_name [id=72f5b283fad34ca3b10e5016af6482cb]). Expected version 3 got 2
I20240502 14:15:14.486198 16308 tablet_copy_service.cc:342] P 5297e816f782430d9a01220f0471a882: Request end of tablet copy session cd67493f6f0142f896010f2b8a39887d-2cb770f077c84cb99985ef59253f133b received from {username='slave'} at 127.6.196.194:56211
I20240502 14:15:14.486307 16308 tablet_copy_service.cc:433] P 5297e816f782430d9a01220f0471a882: ending tablet copy session cd67493f6f0142f896010f2b8a39887d-2cb770f077c84cb99985ef59253f133b on tablet 2cb770f077c84cb99985ef59253f133b with peer cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:14.490764 20160 consensus_queue.cc:1035] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.495321 13186 raft_consensus.cc:1203] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:14.495466 13045 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=72f5b283fad34ca3b10e5016af6482cb] alter complete (version 3)
I20240502 14:15:14.502810 20205 tablet.cc:1722] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:14.509327 20207 tablet_bootstrap.cc:492] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:14.509665 20207 tablet_bootstrap.cc:492] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap complete.
I20240502 14:15:14.511591 20207 ts_tablet_manager.cc:1367] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.134s user 0.023s sys 0.017s
I20240502 14:15:14.512390 20207 raft_consensus.cc:348] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:14.512603 20207 raft_consensus.cc:729] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: LEARNER
I20240502 14:15:14.512784 20207 consensus_queue.cc:260] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:14.513809 13447 raft_consensus.cc:1203] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:14.525882 20207 ts_tablet_manager.cc:1398] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240502 14:15:14.526583 16442 tablet_copy_service.cc:342] P 8b47d4390f8343a493d6c00123785968: Request end of tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-7a740ff5e0964efcb1fd7dd54dc4094c received from {username='slave'} at 127.6.196.196:52755
I20240502 14:15:14.526729 16442 tablet_copy_service.cc:433] P 8b47d4390f8343a493d6c00123785968: ending tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-7a740ff5e0964efcb1fd7dd54dc4094c on tablet 7a740ff5e0964efcb1fd7dd54dc4094c with peer 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:15:14.527628 20266 ts_tablet_manager.cc:902] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:14.538408 20248 tablet_copy_client.cc:308] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.200:43857
I20240502 14:15:14.539008 16442 tablet_copy_service.cc:140] P 8b47d4390f8343a493d6c00123785968: Received BeginTabletCopySession request for tablet 1b34f2a22306459784a1ad7b81cd1683 from peer 7eb4ecb2379b44c0954f2658c4a04350 ({username='slave'} at 127.6.196.196:52755)
I20240502 14:15:14.539177 16442 tablet_copy_service.cc:161] P 8b47d4390f8343a493d6c00123785968: Beginning new tablet copy session on tablet 1b34f2a22306459784a1ad7b81cd1683 from peer 7eb4ecb2379b44c0954f2658c4a04350 at {username='slave'} at 127.6.196.196:52755: session id = 7eb4ecb2379b44c0954f2658c4a04350-1b34f2a22306459784a1ad7b81cd1683
I20240502 14:15:14.540800 20247 tablet_copy_client.cc:308] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:15:14.541419 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet cee69dfc4ded45de81cfd8b06c5f2f2f from peer cd67493f6f0142f896010f2b8a39887d ({username='slave'} at 127.6.196.194:35821)
I20240502 14:15:14.541551 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet cee69dfc4ded45de81cfd8b06c5f2f2f from peer cd67493f6f0142f896010f2b8a39887d at {username='slave'} at 127.6.196.194:35821: session id = cd67493f6f0142f896010f2b8a39887d-cee69dfc4ded45de81cfd8b06c5f2f2f
I20240502 14:15:14.544185 13601 tablet_copy_source_session.cc:205] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:14.545543 16442 tablet_copy_source_session.cc:205] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:14.546617 16285 consensus_queue.cc:237] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:14.547771 20248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b34f2a22306459784a1ad7b81cd1683. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.552426 20247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee69dfc4ded45de81cfd8b06c5f2f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.555152 20248 tablet_copy_client.cc:783] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:14.556018 20248 tablet_copy_client.cc:647] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:14.561940 20248 tablet_copy_client.cc:523] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:14.568101 13580 raft_consensus.cc:1203] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
W20240502 14:15:14.573343 16219 consensus_peers.cc:485] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c251cfdbec94259be5afbb1b7dbaa38. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:14.574255 20248 tablet_bootstrap.cc:492] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:14.574510 16023 raft_consensus.cc:1261] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:14.575006 20056 consensus_queue.cc:1035] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:14.581836 16152 raft_consensus.cc:1261] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:14.582396 20056 consensus_queue.cc:1035] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:14.584308 20266 tablet_copy_client.cc:308] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.197:43695
I20240502 14:15:14.584889 16043 tablet_copy_service.cc:140] P d1da7f11f0604f78934354a992bb0f55: Received BeginTabletCopySession request for tablet eb472259eca84c31b2ff9f33f7e61177 from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:42217)
I20240502 14:15:14.585042 16043 tablet_copy_service.cc:161] P d1da7f11f0604f78934354a992bb0f55: Beginning new tablet copy session on tablet eb472259eca84c31b2ff9f33f7e61177 from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:42217: session id = c86f7440570f4796babfd44c3d7fd847-eb472259eca84c31b2ff9f33f7e61177
I20240502 14:15:14.588050 16043 tablet_copy_source_session.cc:205] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:14.590937 20266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb472259eca84c31b2ff9f33f7e61177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.590772 20056 raft_consensus.cc:2932] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:14.592677 20248 tablet_bootstrap.cc:492] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:14.593077 20248 tablet_bootstrap.cc:492] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap complete.
I20240502 14:15:14.594780 16152 raft_consensus.cc:2932] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:14.595373 20248 ts_tablet_manager.cc:1367] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.003s
I20240502 14:15:14.596030 20248 raft_consensus.cc:348] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:14.596307 20248 raft_consensus.cc:729] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: LEARNER
I20240502 14:15:14.597435 20248 consensus_queue.cc:260] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:14.599076 20266 tablet_copy_client.cc:783] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:14.599354 20266 tablet_copy_client.cc:647] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:14.600698 20248 ts_tablet_manager.cc:1398] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:15:14.600853 20266 tablet_copy_client.cc:523] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:14.601918 16442 tablet_copy_service.cc:342] P 8b47d4390f8343a493d6c00123785968: Request end of tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-1b34f2a22306459784a1ad7b81cd1683 received from {username='slave'} at 127.6.196.196:52755
I20240502 14:15:14.602031 16442 tablet_copy_service.cc:433] P 8b47d4390f8343a493d6c00123785968: ending tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-1b34f2a22306459784a1ad7b81cd1683 on tablet 1b34f2a22306459784a1ad7b81cd1683 with peer 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:15:14.603967 20266 tablet_bootstrap.cc:492] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:14.605374 16020 raft_consensus.cc:2932] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:14.611090 13045 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:14.611327 13045 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:14.614675 13045 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet d587d24c7cc04e268d6124aafd8ff04a
I20240502 14:15:14.614581 13046 catalog_manager.cc:5484] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:14.615540 16265 tablet_service.cc:1514] Processing DeleteTablet for tablet d587d24c7cc04e268d6124aafd8ff04a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:14 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:14.616029 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet d587d24c7cc04e268d6124aafd8ff04a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:14 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:15:14.616189 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet d587d24c7cc04e268d6124aafd8ff04a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:14 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:15:14.627622 20279 tablet_replica.cc:330] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:14.628623 20281 tablet_replica.cc:330] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:15:14.628928 20279 raft_consensus.cc:2229] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:14.632170 20279 raft_consensus.cc:2258] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:14.632462 20281 raft_consensus.cc:2229] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:14.632922 20281 raft_consensus.cc:2258] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:14.634136 20279 ts_tablet_manager.cc:1875] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:14.635089 20281 ts_tablet_manager.cc:1875] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:14.637586 20280 tablet_replica.cc:330] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:15:14.638281 20280 raft_consensus.cc:2229] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:14.638893 20280 raft_consensus.cc:2258] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:14.644372 20249 tablet_bootstrap.cc:492] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:14.645260 20249 tablet_bootstrap.cc:492] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:15:14.645545 20249 ts_tablet_manager.cc:1367] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.189s user 0.047s sys 0.007s
I20240502 14:15:14.646540 20249 raft_consensus.cc:348] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:14.646888 20249 raft_consensus.cc:729] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:15:14.647146 20249 consensus_queue.cc:260] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:14.646215 20280 ts_tablet_manager.cc:1875] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:14.648926 20266 tablet_bootstrap.cc:492] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:14.649082 20249 ts_tablet_manager.cc:1398] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:15:14.649559 20266 tablet_bootstrap.cc:492] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:15:14.649652 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-16c817b864c4425eaa85ac4b5aff59fa received from {username='slave'} at 127.6.196.197:40701
I20240502 14:15:14.649735 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-16c817b864c4425eaa85ac4b5aff59fa on tablet 16c817b864c4425eaa85ac4b5aff59fa with peer d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:14.649775 20266 ts_tablet_manager.cc:1367] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.046s user 0.010s sys 0.017s
I20240502 14:15:14.650499 20266 raft_consensus.cc:348] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:14.652710 20266 raft_consensus.cc:729] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:15:14.652932 20266 consensus_queue.cc:260] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:14.656580 20266 ts_tablet_manager.cc:1398] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:15:14.657263 16043 tablet_copy_service.cc:342] P d1da7f11f0604f78934354a992bb0f55: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-eb472259eca84c31b2ff9f33f7e61177 received from {username='slave'} at 127.6.196.198:42217
I20240502 14:15:14.657403 16043 tablet_copy_service.cc:433] P d1da7f11f0604f78934354a992bb0f55: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-eb472259eca84c31b2ff9f33f7e61177 on tablet eb472259eca84c31b2ff9f33f7e61177 with peer c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:14.659803 20281 ts_tablet_manager.cc:1888] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:14.659951 20281 log.cc:1196] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/d587d24c7cc04e268d6124aafd8ff04a
I20240502 14:15:14.660298 20281 ts_tablet_manager.cc:1909] T d587d24c7cc04e268d6124aafd8ff04a P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:15:14.661738 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet d587d24c7cc04e268d6124aafd8ff04a (table rebalancer_dynamic_table_new_name [id=72f5b283fad34ca3b10e5016af6482cb]) successfully deleted
I20240502 14:15:14.664002 20280 ts_tablet_manager.cc:1888] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:14.664072 20280 log.cc:1196] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/d587d24c7cc04e268d6124aafd8ff04a
I20240502 14:15:14.664311 20280 ts_tablet_manager.cc:1909] T d587d24c7cc04e268d6124aafd8ff04a P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:15:14.665035 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet d587d24c7cc04e268d6124aafd8ff04a (table rebalancer_dynamic_table_new_name [id=72f5b283fad34ca3b10e5016af6482cb]) successfully deleted
I20240502 14:15:14.665930 20247 tablet_copy_client.cc:783] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:14.666148 20247 tablet_copy_client.cc:647] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:14.666282 20279 ts_tablet_manager.cc:1888] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:14.666380 20279 log.cc:1196] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/d587d24c7cc04e268d6124aafd8ff04a
I20240502 14:15:14.666780 20279 ts_tablet_manager.cc:1909] T d587d24c7cc04e268d6124aafd8ff04a P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:15:14.667564 20247 tablet_copy_client.cc:523] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:14.667574 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet d587d24c7cc04e268d6124aafd8ff04a (table rebalancer_dynamic_table_new_name [id=72f5b283fad34ca3b10e5016af6482cb]) successfully deleted
I20240502 14:15:14.669972 20247 tablet_bootstrap.cc:492] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:14.675271 20247 tablet_bootstrap.cc:492] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:14.675633 20247 tablet_bootstrap.cc:492] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d: Bootstrap complete.
I20240502 14:15:14.675988 20247 ts_tablet_manager.cc:1367] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20240502 14:15:14.676731 20247 raft_consensus.cc:348] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:14.676980 20247 raft_consensus.cc:729] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: LEARNER
I20240502 14:15:14.677178 20247 consensus_queue.cc:260] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:14.677860 20247 ts_tablet_manager.cc:1398] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:14.678661 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session cd67493f6f0142f896010f2b8a39887d-cee69dfc4ded45de81cfd8b06c5f2f2f received from {username='slave'} at 127.6.196.194:35821
I20240502 14:15:14.678733 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session cd67493f6f0142f896010f2b8a39887d-cee69dfc4ded45de81cfd8b06c5f2f2f on tablet cee69dfc4ded45de81cfd8b06c5f2f2f with peer cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:14.687669 16285 consensus_queue.cc:237] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:14.700675 13318 raft_consensus.cc:1261] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:14.700765 16219 consensus_peers.cc:485] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 -> Peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Couldn't send request to peer 7eb4ecb2379b44c0954f2658c4a04350. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: caac1154570d4420ae8764ca18667f29. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:14.700788 16023 raft_consensus.cc:1261] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:14.701292 20056 consensus_queue.cc:1035] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:14.701581 19792 consensus_queue.cc:1035] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:14.704798 19959 raft_consensus.cc:2932] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:14.705330 16023 raft_consensus.cc:2932] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:14.712965 18885 consensus_queue.cc:1035] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.715652 13046 catalog_manager.cc:5484] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:14.717657 13045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:14.729933 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 89be9df9b5264b1b81fb5a22953f66d5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5f63b67ca4304d1589b96cd94598d592]), partition=
I20240502 14:15:14.730316 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89be9df9b5264b1b81fb5a22953f66d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.731343 20101 consensus_queue.cc:1035] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.734772 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 89be9df9b5264b1b81fb5a22953f66d5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5f63b67ca4304d1589b96cd94598d592]), partition=
I20240502 14:15:14.735133 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89be9df9b5264b1b81fb5a22953f66d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.737880 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 89be9df9b5264b1b81fb5a22953f66d5 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5f63b67ca4304d1589b96cd94598d592]), partition=
I20240502 14:15:14.738271 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89be9df9b5264b1b81fb5a22953f66d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.744771 20298 tablet_bootstrap.cc:492] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:14.746233 20298 tablet_bootstrap.cc:654] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.748931 13319 raft_consensus.cc:1203] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:14.760986 20301 tablet_bootstrap.cc:492] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:14.763053 20301 tablet_bootstrap.cc:654] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.763886 20298 tablet_bootstrap.cc:492] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:14.764422 20298 ts_tablet_manager.cc:1367] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.004s
I20240502 14:15:14.765117 20298 raft_consensus.cc:348] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:14.765318 20298 raft_consensus.cc:374] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.765400 20298 raft_consensus.cc:729] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.766153 20298 consensus_queue.cc:260] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:14.766611 13318 raft_consensus.cc:2932] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:14.776314 20300 tablet_bootstrap.cc:492] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:14.778775 20300 tablet_bootstrap.cc:654] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.780984 13319 raft_consensus.cc:1203] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:14.787858 20298 ts_tablet_manager.cc:1398] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240502 14:15:14.796838 19951 consensus_queue.cc:1035] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.798130 20300 tablet_bootstrap.cc:492] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:14.798329 20300 ts_tablet_manager.cc:1367] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20240502 14:15:14.799234 20300 raft_consensus.cc:348] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:14.799834 20300 raft_consensus.cc:374] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.800370 20300 raft_consensus.cc:729] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.800722 20300 consensus_queue.cc:260] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:14.801537 20300 ts_tablet_manager.cc:1398] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:14.810597 20301 tablet_bootstrap.cc:492] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:14.810756 20301 ts_tablet_manager.cc:1367] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.000s
I20240502 14:15:14.811589 20301 raft_consensus.cc:348] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:14.811836 20301 raft_consensus.cc:374] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.811887 20301 raft_consensus.cc:729] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.811691 16152 consensus_queue.cc:237] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:14.812117 20301 consensus_queue.cc:260] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:14.812637 20301 ts_tablet_manager.cc:1398] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:14.815019 19791 consensus_queue.cc:1035] T 8a3b25a9e8b049a88bb999d9362e98da P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:14.820986 19951 raft_consensus.cc:1050] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968: attempting to promote NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 to VOTER
I20240502 14:15:14.821389 16023 raft_consensus.cc:1203] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:15:14.821764 19951 consensus_queue.cc:237] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:14.838979 13578 raft_consensus.cc:1261] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:14.839481 13447 raft_consensus.cc:1261] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:14.839901 16023 raft_consensus.cc:1261] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:14.840067 18885 consensus_queue.cc:1035] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:14.840111 20100 consensus_queue.cc:1035] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:14.843497 19791 consensus_queue.cc:1035] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:14.844695 16090 consensus_peers.cc:485] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db2bc203d47c4ca1b3329745a0ff7fe1. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:14.847579 16023 raft_consensus.cc:1261] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:14.848500 16285 raft_consensus.cc:1261] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:14.849941 13046 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:14.851032 20100 consensus_queue.cc:1035] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:14.852485 20100 consensus_queue.cc:1035] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:14.861630 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 939a92466f61423187582d84ab10c0f6 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=176de60994624cffa95f6e64915365f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.861778 16265 tablet_service.cc:1467] Processing CreateTablet for tablet 939a92466f61423187582d84ab10c0f6 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=176de60994624cffa95f6e64915365f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.862058 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939a92466f61423187582d84ab10c0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.862203 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939a92466f61423187582d84ab10c0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.865504 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 939a92466f61423187582d84ab10c0f6 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=176de60994624cffa95f6e64915365f1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.865813 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939a92466f61423187582d84ab10c0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.865737 16399 tablet_service.cc:1467] Processing CreateTablet for tablet eafe2272cca9440ba7060f64106a73cb (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=176de60994624cffa95f6e64915365f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.866044 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eafe2272cca9440ba7060f64106a73cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.868865 16401 tablet_service.cc:1467] Processing CreateTablet for tablet f9e6bf2985714a02a6e59d4c48c0a253 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=176de60994624cffa95f6e64915365f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.869179 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e6bf2985714a02a6e59d4c48c0a253. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.870566 19791 raft_consensus.cc:2932] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:14.873560 20298 tablet_bootstrap.cc:492] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:14.874754 13427 tablet_service.cc:1467] Processing CreateTablet for tablet f9e6bf2985714a02a6e59d4c48c0a253 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=176de60994624cffa95f6e64915365f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.874634 13447 raft_consensus.cc:2932] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:14.875102 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e6bf2985714a02a6e59d4c48c0a253. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.875190 20298 tablet_bootstrap.cc:654] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.879621 20101 raft_consensus.cc:1050] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968: attempting to promote NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 to VOTER
I20240502 14:15:14.880447 20101 consensus_queue.cc:237] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:14.880959 13298 tablet_service.cc:1467] Processing CreateTablet for tablet eafe2272cca9440ba7060f64106a73cb (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=176de60994624cffa95f6e64915365f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.881357 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eafe2272cca9440ba7060f64106a73cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.882189 16267 tablet_service.cc:1467] Processing CreateTablet for tablet f9e6bf2985714a02a6e59d4c48c0a253 (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=176de60994624cffa95f6e64915365f1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.882596 16267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9e6bf2985714a02a6e59d4c48c0a253. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.885916 16266 tablet_service.cc:1467] Processing CreateTablet for tablet eafe2272cca9440ba7060f64106a73cb (DEFAULT_TABLE table=rebalancer_extra_table_46 [id=176de60994624cffa95f6e64915365f1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:14.886276 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eafe2272cca9440ba7060f64106a73cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:14.888664 19951 raft_consensus.cc:2932] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:14.888865 16020 raft_consensus.cc:2932] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:14.890889 13580 raft_consensus.cc:2932] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:14.893028 13446 raft_consensus.cc:1261] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:14.894100 20100 consensus_queue.cc:1035] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:14.896311 19171 raft_consensus.cc:1050] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55: attempting to promote NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 to VOTER
I20240502 14:15:14.897632 13040 catalog_manager.cc:5484] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:14.898170 19171 consensus_queue.cc:237] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:14.899667 13045 catalog_manager.cc:5484] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:14.903126 16285 raft_consensus.cc:1261] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:14.904471 20100 consensus_queue.cc:1035] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:14.918190 20321 tablet_bootstrap.cc:492] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:14.919044 20317 tablet_bootstrap.cc:492] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:14.920397 20321 tablet_bootstrap.cc:654] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.922411 13580 raft_consensus.cc:1203] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240502 14:15:14.924394 16152 raft_consensus.cc:1261] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:14.925917 20100 consensus_queue.cc:1035] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:14.925626 16020 raft_consensus.cc:2932] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:14.926805 20298 tablet_bootstrap.cc:492] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:14.926982 20298 ts_tablet_manager.cc:1367] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.007s
I20240502 14:15:14.927687 20298 raft_consensus.cc:348] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:14.927961 20298 raft_consensus.cc:374] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:14.928088 20298 raft_consensus.cc:729] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:14.928401 20298 consensus_queue.cc:260] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:14.929050 20298 ts_tablet_manager.cc:1398] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:14.932499 20317 tablet_bootstrap.cc:654] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.934613 16285 raft_consensus.cc:1261] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:14.935259 19904 consensus_queue.cc:1035] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:14.939081 20301 tablet_bootstrap.cc:492] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:14.940352 20301 tablet_bootstrap.cc:654] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.941817 13444 raft_consensus.cc:1261] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:14.942332 16419 raft_consensus.cc:1261] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:14.942865 19904 consensus_queue.cc:1035] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:14.944626 19904 consensus_queue.cc:1035] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:14.959240 20298 tablet_bootstrap.cc:492] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:14.960634 20298 tablet_bootstrap.cc:654] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:14.972447 20101 raft_consensus.cc:2932] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:14.978535 16152 raft_consensus.cc:2932] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:14.984146 20074 raft_consensus.cc:2932] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:14.984812 16421 raft_consensus.cc:2932] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:14.989131 16288 raft_consensus.cc:2932] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:14.993304 13445 raft_consensus.cc:2932] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:14.995885 20339 ts_tablet_manager.cc:902] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:15.003330 16285 raft_consensus.cc:2932] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:15.004225 20298 tablet_bootstrap.cc:492] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:15.004370 20298 ts_tablet_manager.cc:1367] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.013s
I20240502 14:15:15.005049 20298 raft_consensus.cc:348] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:15.005326 20298 raft_consensus.cc:374] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.005419 20298 raft_consensus.cc:729] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.005674 20298 consensus_queue.cc:260] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:15.006264 20298 ts_tablet_manager.cc:1398] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:15.017654 20074 raft_consensus.cc:482] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:15.017887 20074 raft_consensus.cc:504] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:15.018769 20074 leader_election.cc:288] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:15.019462 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89be9df9b5264b1b81fb5a22953f66d5" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:15.019688 13187 raft_consensus.cc:2454] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:15.020412 15955 leader_election.cc:302] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:15.020905 19904 raft_consensus.cc:2781] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:15.021015 19904 raft_consensus.cc:482] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:15.021116 19904 raft_consensus.cc:3037] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.021402 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89be9df9b5264b1b81fb5a22953f66d5" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:15.021617 16421 raft_consensus.cc:2454] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:15.024189 19904 raft_consensus.cc:504] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:15.025650 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89be9df9b5264b1b81fb5a22953f66d5" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:15.025868 13187 raft_consensus.cc:3037] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.027609 20338 tablet_bootstrap.cc:492] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:15.032327 13187 raft_consensus.cc:2454] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:15.032752 19904 leader_election.cc:288] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:15.033306 15955 leader_election.cc:302] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:15.033730 20074 raft_consensus.cc:2781] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:15.033890 20074 raft_consensus.cc:686] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:15.034179 20074 consensus_queue.cc:237] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:15.036746 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89be9df9b5264b1b81fb5a22953f66d5" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:15.036993 16421 raft_consensus.cc:3037] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.039788 16421 raft_consensus.cc:2454] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:15.046864 16287 raft_consensus.cc:2932] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.055702 20339 tablet_copy_client.cc:308] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.199:33857
I20240502 14:15:15.056326 16308 tablet_copy_service.cc:140] P 5297e816f782430d9a01220f0471a882: Received BeginTabletCopySession request for tablet 1c251cfdbec94259be5afbb1b7dbaa38 from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:55953)
I20240502 14:15:15.056459 16308 tablet_copy_service.cc:161] P 5297e816f782430d9a01220f0471a882: Beginning new tablet copy session on tablet 1c251cfdbec94259be5afbb1b7dbaa38 from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:55953: session id = 20a967bad5504b1e9c6fb720ecdd1625-1c251cfdbec94259be5afbb1b7dbaa38
I20240502 14:15:15.056906 13043 catalog_manager.cc:5484] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:15.058816 16308 tablet_copy_source_session.cc:205] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:15.059877 20339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c251cfdbec94259be5afbb1b7dbaa38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.062060 13045 catalog_manager.cc:5484] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:15.070267 20339 tablet_copy_client.cc:783] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:15.070515 20339 tablet_copy_client.cc:647] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:15.072203 20339 tablet_copy_client.cc:523] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:15.075390 20339 tablet_bootstrap.cc:492] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:15.082445 16421 consensus_queue.cc:237] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:15.084993 13445 raft_consensus.cc:1261] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.085193 16152 raft_consensus.cc:1261] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.086119 19904 raft_consensus.cc:1050] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55: attempting to promote NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 to VOTER
I20240502 14:15:15.086282 19951 consensus_queue.cc:1035] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.086850 19904 consensus_queue.cc:237] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:15.087388 13445 raft_consensus.cc:2932] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:15.091276 20100 consensus_queue.cc:1035] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.096513 16421 consensus_queue.cc:237] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:15.100065 16287 raft_consensus.cc:1261] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.100966 13580 raft_consensus.cc:1261] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.101131 19951 consensus_queue.cc:1035] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.101866 20100 consensus_queue.cc:1035] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.105207 16155 raft_consensus.cc:1203] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:15.107122 20100 raft_consensus.cc:2932] T c28e61290d744d4c9409947f9e41706e P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.107199 20326 raft_consensus.cc:2932] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:15.108069 20101 raft_consensus.cc:1050] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968: attempting to promote NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 to VOTER
I20240502 14:15:15.108991 20101 consensus_queue.cc:237] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:15.110016 16152 raft_consensus.cc:2932] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:15.114554 20339 tablet_bootstrap.cc:492] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:15.114686 13578 raft_consensus.cc:2932] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.115712 20339 tablet_bootstrap.cc:492] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:15:15.115767 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a1c0b2fdd1d64bac8100d797c86fd8b2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:15.115911 20339 ts_tablet_manager.cc:1367] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.041s user 0.011s sys 0.000s
I20240502 14:15:15.116499 20339 raft_consensus.cc:348] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:15.116693 20339 raft_consensus.cc:729] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:15:15.116835 20339 consensus_queue.cc:260] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:15.119498 13187 raft_consensus.cc:1261] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.120119 20340 consensus_queue.cc:1035] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.121788 16152 raft_consensus.cc:1261] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.122231 20340 consensus_queue.cc:1035] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.128881 13578 raft_consensus.cc:1261] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 LEARNER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:15:15.129465 20311 consensus_queue.cc:1035] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.131127 16152 raft_consensus.cc:1261] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:15:15.131670 20311 consensus_queue.cc:1035] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.135521 16421 raft_consensus.cc:1261] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.136977 19904 consensus_queue.cc:1035] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.137866 16020 raft_consensus.cc:1261] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:15:15.140261 20311 consensus_queue.cc:1035] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.152989 20339 ts_tablet_manager.cc:1398] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.037s user 0.002s sys 0.000s
I20240502 14:15:15.153650 16308 tablet_copy_service.cc:342] P 5297e816f782430d9a01220f0471a882: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-1c251cfdbec94259be5afbb1b7dbaa38 received from {username='slave'} at 127.6.196.193:55953
I20240502 14:15:15.153805 16308 tablet_copy_service.cc:433] P 5297e816f782430d9a01220f0471a882: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-1c251cfdbec94259be5afbb1b7dbaa38 on tablet 1c251cfdbec94259be5afbb1b7dbaa38 with peer 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:15:15.154376 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c28e61290d744d4c9409947f9e41706e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:15.162858 20101 raft_consensus.cc:2932] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.164424 16152 raft_consensus.cc:2932] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.164944 20301 tablet_bootstrap.cc:492] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:15.165160 20301 ts_tablet_manager.cc:1367] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.226s user 0.001s sys 0.011s
I20240502 14:15:15.166316 20301 raft_consensus.cc:348] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:15.166584 20301 raft_consensus.cc:374] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.171041 13580 raft_consensus.cc:2932] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.172340 13038 catalog_manager.cc:5484] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.179316 13046 catalog_manager.cc:5484] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:15.180325 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet c28e61290d744d4c9409947f9e41706e with delete_type TABLET_DATA_TOMBSTONED (TS d1da7f11f0604f78934354a992bb0f55 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:15.182524 20101 raft_consensus.cc:482] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:15.182642 20101 raft_consensus.cc:504] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:15.183820 20101 leader_election.cc:288] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:15.189247 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafe2272cca9440ba7060f64106a73cb" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
W20240502 14:15:15.190083 16355 leader_election.cc:341] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:15.190229 16285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafe2272cca9440ba7060f64106a73cb" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
W20240502 14:15:15.190903 16355 leader_election.cc:341] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:15.191076 16355 leader_election.cc:302] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b47d4390f8343a493d6c00123785968; no voters: 5297e816f782430d9a01220f0471a882, cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:15.191541 20101 raft_consensus.cc:2726] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:15.197691 13445 raft_consensus.cc:2932] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:15.205426 20317 tablet_bootstrap.cc:492] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:15.206760 20317 ts_tablet_manager.cc:1367] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.289s user 0.036s sys 0.030s
I20240502 14:15:15.207495 16265 tablet_service.cc:1514] Processing DeleteTablet for tablet a1c0b2fdd1d64bac8100d797c86fd8b2 with delete_type TABLET_DATA_TOMBSTONED (TS 5297e816f782430d9a01220f0471a882 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:15.208701 20317 raft_consensus.cc:348] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:15.209875 20317 raft_consensus.cc:374] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.210152 20317 raft_consensus.cc:729] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.210675 20317 consensus_queue.cc:260] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:15.212044 20317 ts_tablet_manager.cc:1398] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:15:15.211879 16287 raft_consensus.cc:2932] T c28e61290d744d4c9409947f9e41706e P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.213002 20317 tablet_bootstrap.cc:492] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:15.214862 20317 tablet_bootstrap.cc:654] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.228014 13038 catalog_manager.cc:5484] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.232738 20317 tablet_bootstrap.cc:492] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:15.233547 20317 ts_tablet_manager.cc:1367] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240502 14:15:15.233798 16287 consensus_queue.cc:237] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:15.234877 20317 raft_consensus.cc:348] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:15.236482 20317 raft_consensus.cc:374] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.236789 20317 raft_consensus.cc:729] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.237213 20317 consensus_queue.cc:260] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:15.238569 20301 raft_consensus.cc:729] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.238709 20317 ts_tablet_manager.cc:1398] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:15:15.238914 20301 consensus_queue.cc:260] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:15.239661 20301 ts_tablet_manager.cc:1398] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.074s user 0.000s sys 0.002s
I20240502 14:15:15.241461 20338 tablet_bootstrap.cc:654] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.257633 20359 tablet_replica.cc:330] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:15.257964 20360 tablet_replica.cc:330] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:15.258004 20359 raft_consensus.cc:2229] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:15.258224 20360 raft_consensus.cc:2229] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:15.258422 20359 raft_consensus.cc:2258] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:15.258518 20360 raft_consensus.cc:2258] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:15.259256 20360 ts_tablet_manager.cc:1875] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:15.259738 20321 tablet_bootstrap.cc:492] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:15.259955 20321 ts_tablet_manager.cc:1367] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.342s user 0.034s sys 0.056s
I20240502 14:15:15.260905 20321 raft_consensus.cc:348] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:15.261198 20321 raft_consensus.cc:374] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.261372 20321 raft_consensus.cc:729] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.261713 20321 consensus_queue.cc:260] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:15.264009 20321 ts_tablet_manager.cc:1398] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:15:15.265726 19904 raft_consensus.cc:2932] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:15.266574 20160 raft_consensus.cc:1050] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER cd67493f6f0142f896010f2b8a39887d to VOTER
I20240502 14:15:15.267282 20160 consensus_queue.cc:237] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:15.269632 19613 raft_consensus.cc:1050] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER d1da7f11f0604f78934354a992bb0f55 to VOTER
I20240502 14:15:15.270143 20321 tablet_bootstrap.cc:492] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:15.270087 13040 catalog_manager.cc:5484] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:15.271039 13187 raft_consensus.cc:2932] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:15.272132 20321 tablet_bootstrap.cc:654] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.272079 16020 raft_consensus.cc:2932] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.274899 16419 raft_consensus.cc:2932] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:15.279929 16023 consensus_queue.cc:237] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:15.282431 13446 raft_consensus.cc:1261] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.283002 19904 consensus_queue.cc:1035] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.284358 16419 consensus_queue.cc:237] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:15.285467 20100 raft_consensus.cc:482] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:15.285634 20100 raft_consensus.cc:504] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:15.282255 16288 raft_consensus.cc:1261] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.286202 20100 leader_election.cc:288] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:15.287052 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e6bf2985714a02a6e59d4c48c0a253" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:15.287009 13045 catalog_manager.cc:5484] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
W20240502 14:15:15.287508 16355 leader_election.cc:341] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:15.287706 19904 consensus_queue.cc:1035] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.289930 19171 raft_consensus.cc:2932] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:15.290452 13580 raft_consensus.cc:1261] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:15.290781 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e6bf2985714a02a6e59d4c48c0a253" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:15.291060 20100 consensus_queue.cc:1035] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240502 14:15:15.291290 16353 leader_election.cc:341] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:15.291424 16353 leader_election.cc:302] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b47d4390f8343a493d6c00123785968; no voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:15:15.291720 20100 raft_consensus.cc:2726] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:15.293321 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2363a23cdde54893ae807ef3960dd032 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:15.294458 16288 raft_consensus.cc:2932] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:15.299829 20338 tablet_bootstrap.cc:492] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:15.300112 20338 ts_tablet_manager.cc:1367] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.273s user 0.004s sys 0.000s
I20240502 14:15:15.283260 20360 ts_tablet_manager.cc:1888] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:15.302601 20360 log.cc:1196] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/a1c0b2fdd1d64bac8100d797c86fd8b2
I20240502 14:15:15.303406 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet a1c0b2fdd1d64bac8100d797c86fd8b2 (table rebalancer_extra_table_18 [id=9dc6b07ff15645ab9222cc55c5a4a00b]) successfully deleted
I20240502 14:15:15.304134 13038 catalog_manager.cc:5484] T 2363a23cdde54893ae807ef3960dd032 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:15.309463 16022 consensus_queue.cc:237] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:15.312048 13187 raft_consensus.cc:1261] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.312664 19171 consensus_queue.cc:1035] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.313205 20321 tablet_bootstrap.cc:492] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:15.313388 20359 ts_tablet_manager.cc:1875] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:15.313496 20321 ts_tablet_manager.cc:1367] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.003s
I20240502 14:15:15.314698 16419 raft_consensus.cc:1261] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.314620 20321 raft_consensus.cc:348] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:15.314834 19613 consensus_queue.cc:237] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:15.315016 20321 raft_consensus.cc:374] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.315268 20321 raft_consensus.cc:729] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.315711 20321 consensus_queue.cc:260] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:15.316480 20321 ts_tablet_manager.cc:1398] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:15.316885 20321 tablet_bootstrap.cc:492] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:15.319555 20321 tablet_bootstrap.cc:654] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:15.321409 16155 raft_consensus.cc:1261] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:15.322345 16419 raft_consensus.cc:1261] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:15.323180 16219 consensus_peers.cc:485] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c78403038b614e6db2afb617328f6411. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:15.323338 20056 consensus_queue.cc:1035] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240502 14:15:15.323904 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet 2363a23cdde54893ae807ef3960dd032 with delete_type TABLET_DATA_TOMBSTONED (TS 8b47d4390f8343a493d6c00123785968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:15.324160 19959 consensus_queue.cc:1035] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:15.330338 13319 raft_consensus.cc:1261] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.334072 20162 consensus_queue.cc:1035] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.336827 20367 ts_tablet_manager.cc:902] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350: Initiating tablet copy from peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:15.349399 20100 raft_consensus.cc:2932] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.355042 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7a740ff5e0964efcb1fd7dd54dc4094c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:15.357075 13186 raft_consensus.cc:1203] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:15.359906 13039 catalog_manager.cc:5484] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New cstate: current_term: 2 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:15.366693 20371 tablet_replica.cc:330] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:15.367729 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a740ff5e0964efcb1fd7dd54dc4094c with delete_type TABLET_DATA_TOMBSTONED (TS c86f7440570f4796babfd44c3d7fd847 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:15.368470 20383 tablet_replica.cc:330] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:15.368711 20383 raft_consensus.cc:2229] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:15.369078 20383 raft_consensus.cc:2258] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:15.369608 13578 raft_consensus.cc:2932] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.370148 20371 raft_consensus.cc:2229] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:15.370368 20371 raft_consensus.cc:2258] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:15.371126 19959 raft_consensus.cc:1050] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882: attempting to promote NON_VOTER cd67493f6f0142f896010f2b8a39887d to VOTER
I20240502 14:15:15.371920 19959 consensus_queue.cc:237] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:15.372542 20383 ts_tablet_manager.cc:1875] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:15.373095 20371 ts_tablet_manager.cc:1875] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:15.382666 20383 ts_tablet_manager.cc:1888] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240502 14:15:15.382839 20383 log.cc:1196] T 7a740ff5e0964efcb1fd7dd54dc4094c P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/7a740ff5e0964efcb1fd7dd54dc4094c
I20240502 14:15:15.383708 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 7a740ff5e0964efcb1fd7dd54dc4094c (table rebalancer_extra_table_26 [id=a2fc7359aefc4890a033bb4d687e45ff]) successfully deleted
I20240502 14:15:15.383807 20321 tablet_bootstrap.cc:492] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:15.384008 20321 ts_tablet_manager.cc:1367] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.003s
I20240502 14:15:15.384711 20321 raft_consensus.cc:348] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:15.384894 20321 raft_consensus.cc:374] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.385003 20321 raft_consensus.cc:729] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.385236 20321 consensus_queue.cc:260] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:15.385834 20321 ts_tablet_manager.cc:1398] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:15.387534 19391 raft_consensus.cc:482] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:15.387678 19391 raft_consensus.cc:504] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:15.388345 19391 leader_election.cc:288] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:15.389016 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939a92466f61423187582d84ab10c0f6" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:15.389786 13187 raft_consensus.cc:2454] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:15.390353 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939a92466f61423187582d84ab10c0f6" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:15.390617 16288 raft_consensus.cc:2454] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:15.390992 13251 leader_election.cc:302] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:15.393474 19391 raft_consensus.cc:2781] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:15.393568 19391 raft_consensus.cc:482] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:15.393637 19391 raft_consensus.cc:3037] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.394071 20338 raft_consensus.cc:348] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:15.394361 20338 raft_consensus.cc:374] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:15.394474 20338 raft_consensus.cc:729] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:15.394697 20338 consensus_queue.cc:260] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:15.395709 20371 ts_tablet_manager.cc:1888] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:15.395833 20371 log.cc:1196] T 2363a23cdde54893ae807ef3960dd032 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/2363a23cdde54893ae807ef3960dd032
I20240502 14:15:15.396718 19391 raft_consensus.cc:504] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:15.397666 16288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939a92466f61423187582d84ab10c0f6" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:15.397859 16288 raft_consensus.cc:3037] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.398864 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 2363a23cdde54893ae807ef3960dd032 (table rebalancer_extra_table_36 [id=8f0281d7a3a049329cb3130e1f8d0c04]) successfully deleted
I20240502 14:15:15.399822 20338 ts_tablet_manager.cc:1398] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:15:15.400403 16288 raft_consensus.cc:2454] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:15.401120 13253 leader_election.cc:302] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:15.401407 20087 raft_consensus.cc:2781] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:15.300503 13446 raft_consensus.cc:2932] T 2363a23cdde54893ae807ef3960dd032 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:15.360316 13445 raft_consensus.cc:1261] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.409435 20368 raft_consensus.cc:482] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:15.409559 20368 raft_consensus.cc:504] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:15.410198 20368 leader_election.cc:288] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:15.410951 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e6bf2985714a02a6e59d4c48c0a253" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:15.410984 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e6bf2985714a02a6e59d4c48c0a253" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:15.316176 20340 consensus_queue.cc:1035] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.411185 13445 raft_consensus.cc:2454] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:15.411202 16421 raft_consensus.cc:2454] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:15.329918 16018 raft_consensus.cc:1261] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.411816 16219 leader_election.cc:302] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:15.412369 20056 raft_consensus.cc:2781] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:15.412467 20056 raft_consensus.cc:482] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:15.412571 20056 raft_consensus.cc:3037] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.412907 20074 raft_consensus.cc:2932] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:15.413339 13187 raft_consensus.cc:2932] T 154fe82369ef4d2bb38e8f14637463bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:15.414867 20056 raft_consensus.cc:504] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:15.415447 20056 leader_election.cc:288] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:15.416508 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 154fe82369ef4d2bb38e8f14637463bb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:15.417469 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e6bf2985714a02a6e59d4c48c0a253" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:15.417516 16421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9e6bf2985714a02a6e59d4c48c0a253" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:15.417611 13445 raft_consensus.cc:3037] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.417692 16421 raft_consensus.cc:3037] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.420089 13445 raft_consensus.cc:2454] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:15.420807 16023 raft_consensus.cc:1261] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:15.421262 16421 raft_consensus.cc:2454] T f9e6bf2985714a02a6e59d4c48c0a253 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:15.422194 16220 leader_election.cc:302] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:15.422683 20368 raft_consensus.cc:482] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:15.422855 20368 raft_consensus.cc:504] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:15.423519 20368 leader_election.cc:288] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:15.423676 20368 raft_consensus.cc:2781] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:15.423645 20100 consensus_queue.cc:1035] T 7a740ff5e0964efcb1fd7dd54dc4094c P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.423774 20368 raft_consensus.cc:686] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:15.424005 20368 consensus_queue.cc:237] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:15.425182 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafe2272cca9440ba7060f64106a73cb" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:15.425386 16420 raft_consensus.cc:2454] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:15.425128 13039 catalog_manager.cc:5484] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:15.425719 16220 leader_election.cc:302] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:15.426034 16023 raft_consensus.cc:2932] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.427485 16420 raft_consensus.cc:2932] T 154fe82369ef4d2bb38e8f14637463bb P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:15.430191 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafe2272cca9440ba7060f64106a73cb" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:15.430362 13319 raft_consensus.cc:2454] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:15.430650 16135 tablet_service.cc:1514] Processing DeleteTablet for tablet 154fe82369ef4d2bb38e8f14637463bb with delete_type TABLET_DATA_TOMBSTONED (TS c86f7440570f4796babfd44c3d7fd847 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:15.431058 20383 tablet_replica.cc:330] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:15.431324 20383 raft_consensus.cc:2229] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:15.431586 20383 pending_rounds.cc:62] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847: Trying to abort 1 pending ops.
I20240502 14:15:15.431648 20383 pending_rounds.cc:68] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023244554593914880 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "154fe82369ef4d2bb38e8f14637463bb" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } } }
I20240502 14:15:15.432096 20383 raft_consensus.cc:2894] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 14:15:15.432209 20383 raft_consensus.cc:2866] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:15:15.432322 20383 raft_consensus.cc:2258] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:15.433070 20383 ts_tablet_manager.cc:1875] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:15.435209 20056 raft_consensus.cc:2781] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:15.435333 20056 raft_consensus.cc:482] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:15.435448 20056 raft_consensus.cc:3037] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.439450 20383 ts_tablet_manager.cc:1888] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:15.439580 20383 log.cc:1196] T 154fe82369ef4d2bb38e8f14637463bb P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/154fe82369ef4d2bb38e8f14637463bb
I20240502 14:15:15.439910 13040 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:15.440415 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 154fe82369ef4d2bb38e8f14637463bb (table rebalancer_extra_table_32 [id=6cab90f7f1c84cf2b82ee344af1b6617]) successfully deleted
I20240502 14:15:15.447000 13188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "939a92466f61423187582d84ab10c0f6" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:15.447233 13188 raft_consensus.cc:3037] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.450407 13188 raft_consensus.cc:2454] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:15.454360 20087 raft_consensus.cc:686] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:15.457563 20087 consensus_queue.cc:237] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:15.459760 19391 leader_election.cc:288] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
W20240502 14:15:15.460728 13515 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.200:43857, user_credentials={real_user=slave}} blocked reactor thread for 100036us
I20240502 14:15:15.462225 16152 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:15.462805 20375 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:15.463562 20056 raft_consensus.cc:504] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:15.464229 20056 leader_election.cc:288] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
W20240502 14:15:15.474783 13516 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.197:43695, user_credentials={real_user=slave}} blocked reactor thread for 136417us
I20240502 14:15:15.476209 20162 consensus_queue.cc:1035] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.490525 20162 consensus_queue.cc:1035] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.492533 16023 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:15.497678 13186 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:15.511060 20162 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:15.515522 13186 raft_consensus.cc:1261] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:15.515858 16420 raft_consensus.cc:1261] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:15.516214 19171 consensus_queue.cc:1035] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:15.517984 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafe2272cca9440ba7060f64106a73cb" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:15.518124 13317 raft_consensus.cc:1261] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.518158 13318 raft_consensus.cc:3037] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.519115 20100 raft_consensus.cc:1050] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968: attempting to promote NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 to VOTER
I20240502 14:15:15.519840 20100 consensus_queue.cc:237] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:15.520828 13318 raft_consensus.cc:2454] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:15.521824 16221 leader_election.cc:302] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:15.521996 20340 consensus_queue.cc:1035] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240502 14:15:15.523175 20056 consensus_queue.cc:1035] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.525439 20395 raft_consensus.cc:2781] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:15.525595 20395 raft_consensus.cc:686] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:15.527076 20395 consensus_queue.cc:237] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:15.535257 20162 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:15.535369 20359 ts_tablet_manager.cc:1888] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:15.535485 20359 log.cc:1196] T c28e61290d744d4c9409947f9e41706e P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/c28e61290d744d4c9409947f9e41706e
I20240502 14:15:15.540776 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet c28e61290d744d4c9409947f9e41706e (table rebalancer_extra_table_20 [id=35b2e9aa3c11478e9023ef811af72c1a]) successfully deleted
I20240502 14:15:15.557493 20377 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:15.566390 13040 catalog_manager.cc:5484] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:15.568176 16155 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:15.584998 13188 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:15.608431 19171 raft_consensus.cc:1050] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55: attempting to promote NON_VOTER c86f7440570f4796babfd44c3d7fd847 to VOTER
I20240502 14:15:15.608525 20385 ts_tablet_manager.cc:902] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:15.609416 19171 consensus_queue.cc:237] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:15.612844 20162 raft_consensus.cc:2932] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.620805 16020 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:15.622016 13446 raft_consensus.cc:2932] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.624140 13580 consensus_queue.cc:237] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:15.624447 16018 raft_consensus.cc:2932] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.625027 13578 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:15.625829 13317 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20240502 14:15:15.625874 16220 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.196.200:43857, user_credentials={real_user=slave}} blocked reactor thread for 200168us
I20240502 14:15:15.626374 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eafe2272cca9440ba7060f64106a73cb" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:15.626574 16419 raft_consensus.cc:3037] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:15.628885 16419 raft_consensus.cc:2454] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:15.629350 13040 catalog_manager.cc:5484] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.629709 16288 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240502 14:15:15.630211 20326 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:15.630406 16020 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:15.630739 20326 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:15.631139 20374 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:15.631316 20326 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:15.632699 13038 catalog_manager.cc:5484] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:15.621646 13319 raft_consensus.cc:2932] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.641471 16155 raft_consensus.cc:1261] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:15.642535 13039 catalog_manager.cc:5484] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:15.644510 16285 raft_consensus.cc:1261] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.645124 20340 consensus_queue.cc:1035] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.648723 13040 catalog_manager.cc:5484] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:15.651981 16155 raft_consensus.cc:1261] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.652499 20340 consensus_queue.cc:1035] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.658406 13040 catalog_manager.cc:5576] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687) does not have the latest schema for tablet 89be9df9b5264b1b81fb5a22953f66d5 (table rebalancer_dynamic_table [id=5f63b67ca4304d1589b96cd94598d592]). Expected version 1 got 0
I20240502 14:15:15.659377 20369 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:15.662959 16419 raft_consensus.cc:1261] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.663553 20340 consensus_queue.cc:1035] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.663892 16018 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:15.665382 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 16c817b864c4425eaa85ac4b5aff59fa with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:15.665380 20369 consensus_queue.cc:1035] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:15.666530 16152 raft_consensus.cc:2932] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:15.668426 13578 consensus_queue.cc:237] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:15.671751 16018 raft_consensus.cc:1261] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.672307 20369 consensus_queue.cc:1035] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.674278 13038 catalog_manager.cc:5484] T 16c817b864c4425eaa85ac4b5aff59fa P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 8 to 9, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:15.675596 20374 raft_consensus.cc:2932] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.676148 16020 raft_consensus.cc:2932] T cee69dfc4ded45de81cfd8b06c5f2f2f P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.680711 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cee69dfc4ded45de81cfd8b06c5f2f2f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:15.688498 19171 raft_consensus.cc:2932] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:15.694649 16421 raft_consensus.cc:2932] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:15.696305 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c817b864c4425eaa85ac4b5aff59fa with delete_type TABLET_DATA_TOMBSTONED (TS 20a967bad5504b1e9c6fb720ecdd1625 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37690
I20240502 14:15:15.701076 16155 raft_consensus.cc:2932] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:15.706593 13039 catalog_manager.cc:5484] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 3 to 4, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:15.708946 13315 raft_consensus.cc:1261] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.712982 20162 consensus_queue.cc:1035] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.715580 13045 catalog_manager.cc:5484] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:15.716444 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet cee69dfc4ded45de81cfd8b06c5f2f2f with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:15.717921 13315 raft_consensus.cc:2932] T cee69dfc4ded45de81cfd8b06c5f2f2f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.719447 20417 tablet_replica.cc:330] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:15:15.719621 20417 raft_consensus.cc:2229] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:15.719861 20417 raft_consensus.cc:2258] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:15.721879 20418 tablet_replica.cc:330] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:15.722085 16023 raft_consensus.cc:1261] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.722286 20418 raft_consensus.cc:2229] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:15.722649 20418 raft_consensus.cc:2258] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:15.722592 20394 consensus_queue.cc:1035] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.723120 20417 ts_tablet_manager.cc:1875] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:15.724027 20418 ts_tablet_manager.cc:1875] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:15.724149 16419 raft_consensus.cc:1261] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.734246 20420 tablet.cc:1722] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:15.735067 20420 tablet.cc:1620] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:15:15.735558 20421 tablet.cc:1722] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:15.736006 20421 tablet.cc:1620] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:15:15.736779 16020 consensus_queue.cc:237] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:15.738046 16287 raft_consensus.cc:2932] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:15.738826 16152 raft_consensus.cc:1261] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.739362 20074 consensus_queue.cc:1035] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.742643 20417 ts_tablet_manager.cc:1888] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:15:15.743386 20418 ts_tablet_manager.cc:1888] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:15.743501 20418 log.cc:1196] T cee69dfc4ded45de81cfd8b06c5f2f2f P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/cee69dfc4ded45de81cfd8b06c5f2f2f
I20240502 14:15:15.745111 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet cee69dfc4ded45de81cfd8b06c5f2f2f (table rebalancer_extra_table_28 [id=fc31add041984706b1347b9ccdaecbaf]) successfully deleted
I20240502 14:15:15.745653 19904 raft_consensus.cc:2932] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:15.746132 16152 raft_consensus.cc:2932] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:15.749598 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet eb472259eca84c31b2ff9f33f7e61177 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:15.752063 20417 log.cc:1196] T 16c817b864c4425eaa85ac4b5aff59fa P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/16c817b864c4425eaa85ac4b5aff59fa
I20240502 14:15:15.753625 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 16c817b864c4425eaa85ac4b5aff59fa (table rebalance_test_table_7 [id=fe675f309a11456aaf1fc329cfdb1707]) successfully deleted
I20240502 14:15:15.753413 13039 catalog_manager.cc:5484] T eb472259eca84c31b2ff9f33f7e61177 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:15.757022 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet eb472259eca84c31b2ff9f33f7e61177 with delete_type TABLET_DATA_TOMBSTONED (TS 8b47d4390f8343a493d6c00123785968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:15.757457 20371 tablet_replica.cc:330] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:15.757764 20371 raft_consensus.cc:2229] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:15.758014 20371 raft_consensus.cc:2258] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:15.758649 13040 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=5f63b67ca4304d1589b96cd94598d592] alter complete (version 1)
I20240502 14:15:15.758731 20371 ts_tablet_manager.cc:1875] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:15.759122 16285 raft_consensus.cc:1261] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.759666 20367 tablet_copy_client.cc:308] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.199:33857
I20240502 14:15:15.762032 19959 raft_consensus.cc:2932] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:15.763339 19904 consensus_queue.cc:1035] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.763572 16308 tablet_copy_service.cc:140] P 5297e816f782430d9a01220f0471a882: Received BeginTabletCopySession request for tablet caac1154570d4420ae8764ca18667f29 from peer 7eb4ecb2379b44c0954f2658c4a04350 ({username='slave'} at 127.6.196.196:51405)
I20240502 14:15:15.763717 16308 tablet_copy_service.cc:161] P 5297e816f782430d9a01220f0471a882: Beginning new tablet copy session on tablet caac1154570d4420ae8764ca18667f29 from peer 7eb4ecb2379b44c0954f2658c4a04350 at {username='slave'} at 127.6.196.196:51405: session id = 7eb4ecb2379b44c0954f2658c4a04350-caac1154570d4420ae8764ca18667f29
I20240502 14:15:15.765059 16287 raft_consensus.cc:2932] T eb472259eca84c31b2ff9f33f7e61177 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:15.766306 16308 tablet_copy_source_session.cc:205] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:15.768530 20367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caac1154570d4420ae8764ca18667f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:15.773669 20371 ts_tablet_manager.cc:1888] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:15.776376 20394 consensus_queue.cc:1035] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.776552 13039 catalog_manager.cc:5484] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:15.783905 20371 log.cc:1196] T eb472259eca84c31b2ff9f33f7e61177 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/eb472259eca84c31b2ff9f33f7e61177
I20240502 14:15:15.784646 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet eb472259eca84c31b2ff9f33f7e61177 (table rebalancer_extra_table_36 [id=8f0281d7a3a049329cb3130e1f8d0c04]) successfully deleted
I20240502 14:15:15.788095 20394 raft_consensus.cc:2932] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.788726 13318 raft_consensus.cc:2932] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.795420 13038 catalog_manager.cc:5484] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.798018 16419 raft_consensus.cc:2932] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:15.799090 16023 raft_consensus.cc:2932] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.809180 13038 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet 89be9df9b5264b1b81fb5a22953f66d5 (table rebalancer_dynamic_table [id=5f63b67ca4304d1589b96cd94598d592]). Expected version 1 got 0
I20240502 14:15:15.823141 20426 tablet.cc:1722] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:15.826390 13038 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet 89be9df9b5264b1b81fb5a22953f66d5 (table rebalancer_dynamic_table [id=5f63b67ca4304d1589b96cd94598d592]). Expected version 1 got 0
I20240502 14:15:15.829854 20426 tablet.cc:1620] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:15.838317 20100 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.841459 13318 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.843554 13038 catalog_manager.cc:5484] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 5 to 6, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:15.845877 16155 raft_consensus.cc:2932] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:15.848093 16288 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.848702 16421 consensus_queue.cc:237] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:15.850509 13580 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:15.851122 20100 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:15.852332 13580 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.857257 20100 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.864370 20394 consensus_queue.cc:1035] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:15.869117 13319 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:15.872187 13038 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:15.872438 20395 raft_consensus.cc:1050] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882: attempting to promote NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 to VOTER
I20240502 14:15:15.872494 20326 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:15.873135 20395 consensus_queue.cc:237] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:15.874573 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1b34f2a22306459784a1ad7b81cd1683 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:15.877132 13040 catalog_manager.cc:5484] T 1b34f2a22306459784a1ad7b81cd1683 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 6 to 7, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:15.881131 16265 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b34f2a22306459784a1ad7b81cd1683 with delete_type TABLET_DATA_TOMBSTONED (TS 5297e816f782430d9a01220f0471a882 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:15.888267 16287 consensus_queue.cc:237] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:15.888864 20367 tablet_copy_client.cc:783] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:15.889083 20367 tablet_copy_client.cc:647] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:15.890513 20367 tablet_copy_client.cc:523] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:15.893723 20367 tablet_bootstrap.cc:492] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:15.893440 13318 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.893014 13578 raft_consensus.cc:2932] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:15.910959 13319 raft_consensus.cc:1261] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.911152 20420 tablet.cc:1722] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:15.911528 20394 consensus_queue.cc:1035] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.911741 20420 tablet.cc:1620] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:15:15.913980 20056 raft_consensus.cc:2932] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.919219 20421 tablet.cc:1722] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:15.919680 20421 tablet.cc:1620] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: MemRowSet was empty: no flush needed.
I20240502 14:15:15.922361 20438 ts_tablet_manager.cc:902] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3: Initiating tablet copy from peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:15.923166 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2cb770f077c84cb99985ef59253f133b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:15.924177 13319 raft_consensus.cc:2932] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.929901 13045 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=5f63b67ca4304d1589b96cd94598d592] alter complete (version 2)
I20240502 14:15:15.949681 13186 raft_consensus.cc:1261] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.950330 20368 consensus_queue.cc:1035] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.955997 16020 raft_consensus.cc:1261] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:15.956430 13039 catalog_manager.cc:5484] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.956670 20368 consensus_queue.cc:1035] T 2cb770f077c84cb99985ef59253f133b P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:15.958336 16020 raft_consensus.cc:2932] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:15.960327 16023 raft_consensus.cc:1261] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.960834 20368 consensus_queue.cc:1035] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.971921 20435 tablet_replica.cc:330] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:15.972172 20435 raft_consensus.cc:2229] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:15.972437 20435 raft_consensus.cc:2258] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:15.973407 20435 ts_tablet_manager.cc:1875] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:15.973850 20385 tablet_copy_client.cc:308] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.198:42041
I20240502 14:15:15.978621 16155 raft_consensus.cc:1261] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:15.981213 20435 ts_tablet_manager.cc:1888] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240502 14:15:15.980728 20426 tablet.cc:1722] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:15.981465 20435 log.cc:1196] T 1b34f2a22306459784a1ad7b81cd1683 P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/1b34f2a22306459784a1ad7b81cd1683
I20240502 14:15:15.981740 20426 tablet.cc:1620] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:15.982270 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 1b34f2a22306459784a1ad7b81cd1683 (table rebalancer_extra_table_14 [id=71087d0bec7e450da33bd69a7f555355]) successfully deleted
I20240502 14:15:15.984612 19959 consensus_queue.cc:1035] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:15.985056 19959 consensus_queue.cc:1035] T f9e6bf2985714a02a6e59d4c48c0a253 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:15.985813 19959 consensus_queue.cc:1035] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:15.985880 16175 tablet_copy_service.cc:140] P c86f7440570f4796babfd44c3d7fd847: Received BeginTabletCopySession request for tablet db2bc203d47c4ca1b3329745a0ff7fe1 from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:32813)
I20240502 14:15:15.986063 16175 tablet_copy_service.cc:161] P c86f7440570f4796babfd44c3d7fd847: Beginning new tablet copy session on tablet db2bc203d47c4ca1b3329745a0ff7fe1 from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:32813: session id = 5297e816f782430d9a01220f0471a882-db2bc203d47c4ca1b3329745a0ff7fe1
I20240502 14:15:15.989837 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cb770f077c84cb99985ef59253f133b with delete_type TABLET_DATA_TOMBSTONED (TS 8b47d4390f8343a493d6c00123785968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:15.990180 20371 tablet_replica.cc:330] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:15.990417 20371 raft_consensus.cc:2229] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:15.990573 20371 pending_rounds.cc:62] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968: Trying to abort 1 pending ops.
I20240502 14:15:15.990689 20371 pending_rounds.cc:68] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023244555761336320 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "2cb770f077c84cb99985ef59253f133b" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } } }
I20240502 14:15:15.991089 20371 raft_consensus.cc:2894] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240502 14:15:15.991204 20371 raft_consensus.cc:2866] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240502 14:15:15.991303 20371 raft_consensus.cc:2258] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:15.992053 20371 ts_tablet_manager.cc:1875] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:15.993480 16175 tablet_copy_source_session.cc:205] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:15.994500 20087 consensus_queue.cc:1035] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.002445 20438 tablet_copy_client.cc:308] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.199:33857
I20240502 14:15:16.006518 20371 ts_tablet_manager.cc:1888] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:16.006707 20371 log.cc:1196] T 2cb770f077c84cb99985ef59253f133b P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/2cb770f077c84cb99985ef59253f133b
I20240502 14:15:16.007112 19959 consensus_queue.cc:1035] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.016714 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 2cb770f077c84cb99985ef59253f133b (table rebalancer_extra_table_20 [id=35b2e9aa3c11478e9023ef811af72c1a]) successfully deleted
I20240502 14:15:16.030804 16308 tablet_copy_service.cc:140] P 5297e816f782430d9a01220f0471a882: Received BeginTabletCopySession request for tablet c78403038b614e6db2afb617328f6411 from peer 88b77f6edc1c4a72b548bf47c05704f3 ({username='slave'} at 127.6.196.195:53361)
I20240502 14:15:16.030942 16308 tablet_copy_service.cc:161] P 5297e816f782430d9a01220f0471a882: Beginning new tablet copy session on tablet c78403038b614e6db2afb617328f6411 from peer 88b77f6edc1c4a72b548bf47c05704f3 at {username='slave'} at 127.6.196.195:53361: session id = 88b77f6edc1c4a72b548bf47c05704f3-c78403038b614e6db2afb617328f6411
I20240502 14:15:16.034548 16308 tablet_copy_source_session.cc:205] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:16.041589 20087 consensus_queue.cc:1035] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.048619 20385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2bc203d47c4ca1b3329745a0ff7fe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.048673 20438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c78403038b614e6db2afb617328f6411. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.054849 20391 raft_consensus.cc:2932] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:16.055507 20438 tablet_copy_client.cc:783] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:16.055683 20438 tablet_copy_client.cc:647] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:16.067212 20438 tablet_copy_client.cc:523] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:16.069984 20438 tablet_bootstrap.cc:492] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:16.070181 20385 tablet_copy_client.cc:783] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:16.070454 20385 tablet_copy_client.cc:647] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:16.074944 13045 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:16.075960 16155 raft_consensus.cc:2932] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:16.076498 16023 raft_consensus.cc:2932] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:16.081959 20385 tablet_copy_client.cc:523] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:16.082386 20367 tablet_bootstrap.cc:492] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:16.082831 20367 tablet_bootstrap.cc:492] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap complete.
I20240502 14:15:16.083067 20367 ts_tablet_manager.cc:1367] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.189s user 0.014s sys 0.057s
I20240502 14:15:16.083600 20367 raft_consensus.cc:348] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:16.083775 20367 raft_consensus.cc:729] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: LEARNER
I20240502 14:15:16.083961 20367 consensus_queue.cc:260] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:16.084679 20385 tablet_bootstrap.cc:492] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:16.086896 20367 ts_tablet_manager.cc:1398] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:15:16.089672 16308 tablet_copy_service.cc:342] P 5297e816f782430d9a01220f0471a882: Request end of tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-caac1154570d4420ae8764ca18667f29 received from {username='slave'} at 127.6.196.196:51405
I20240502 14:15:16.089875 16308 tablet_copy_service.cc:433] P 5297e816f782430d9a01220f0471a882: ending tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-caac1154570d4420ae8764ca18667f29 on tablet caac1154570d4420ae8764ca18667f29 with peer 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:15:16.092762 20420 tablet.cc:1722] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:16.098619 13040 catalog_manager.cc:5484] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:16.107126 20421 tablet.cc:1722] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:16.107331 16287 consensus_queue.cc:237] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:16.116400 13038 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=5f63b67ca4304d1589b96cd94598d592] alter complete (version 3)
I20240502 14:15:16.066462 13186 raft_consensus.cc:2932] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:16.128388 13039 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet 89be9df9b5264b1b81fb5a22953f66d5 (table rebalancer_dynamic_table_new_name [id=5f63b67ca4304d1589b96cd94598d592]). Expected version 3 got 2
I20240502 14:15:16.128973 20426 tablet.cc:1722] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:16.129707 16152 raft_consensus.cc:1261] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:16.137132 20395 consensus_queue.cc:1035] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:16.139372 20438 tablet_bootstrap.cc:492] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:16.139909 20438 tablet_bootstrap.cc:492] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap complete.
I20240502 14:15:16.140228 20438 ts_tablet_manager.cc:1367] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.070s user 0.007s sys 0.000s
I20240502 14:15:16.141016 20438 raft_consensus.cc:348] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:16.154651 13187 raft_consensus.cc:1261] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:16.155328 20395 consensus_queue.cc:1035] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:16.158352 13045 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:16.154259 13446 consensus_queue.cc:237] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:16.166718 20391 raft_consensus.cc:2932] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:16.173213 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6760f0772449088e93bc1eab5e2ab3 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=dbe0387864724576879d3ccdf6c2add8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:16.173671 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6760f0772449088e93bc1eab5e2ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.174679 13040 catalog_manager.cc:5484] T 1c251cfdbec94259be5afbb1b7dbaa38 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:16.175889 20385 tablet_bootstrap.cc:492] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:16.176328 20385 tablet_bootstrap.cc:492] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882: Bootstrap complete.
I20240502 14:15:16.176537 20385 ts_tablet_manager.cc:1367] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.001s
I20240502 14:15:16.177279 20385 raft_consensus.cc:348] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:16.177464 20385 raft_consensus.cc:729] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: LEARNER
I20240502 14:15:16.177593 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 274c6eb091a742089999cc39ad6dd76d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=dbe0387864724576879d3ccdf6c2add8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:16.177649 20385 consensus_queue.cc:260] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:16.177958 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274c6eb091a742089999cc39ad6dd76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.178107 20385 ts_tablet_manager.cc:1398] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:15:16.178479 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c251cfdbec94259be5afbb1b7dbaa38 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:16.179315 16000 tablet_service.cc:1467] Processing CreateTablet for tablet 274c6eb091a742089999cc39ad6dd76d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=dbe0387864724576879d3ccdf6c2add8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:16.179728 16000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274c6eb091a742089999cc39ad6dd76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.180220 20438 raft_consensus.cc:729] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: LEARNER
I20240502 14:15:16.180692 13578 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:16.180855 20438 consensus_queue.cc:260] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:16.181288 19413 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:16.182929 16401 tablet_service.cc:1467] Processing CreateTablet for tablet 274c6eb091a742089999cc39ad6dd76d (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=dbe0387864724576879d3ccdf6c2add8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:16.183329 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274c6eb091a742089999cc39ad6dd76d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.183435 16419 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:16.184870 19413 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:16.192976 13429 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6760f0772449088e93bc1eab5e2ab3 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=dbe0387864724576879d3ccdf6c2add8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:16.193326 13429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6760f0772449088e93bc1eab5e2ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.193852 16399 tablet_service.cc:1467] Processing CreateTablet for tablet d1e9c049ffe74f20b569b9ac8f8a7592 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=dbe0387864724576879d3ccdf6c2add8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:16.194162 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e9c049ffe74f20b569b9ac8f8a7592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.197593 20438 ts_tablet_manager.cc:1398] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.057s user 0.000s sys 0.001s
I20240502 14:15:16.198443 13559 tablet_service.cc:1467] Processing CreateTablet for tablet d1e9c049ffe74f20b569b9ac8f8a7592 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=dbe0387864724576879d3ccdf6c2add8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:16.198802 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e9c049ffe74f20b569b9ac8f8a7592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.202479 15999 tablet_service.cc:1467] Processing CreateTablet for tablet d1e9c049ffe74f20b569b9ac8f8a7592 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=dbe0387864724576879d3ccdf6c2add8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:16.202967 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e9c049ffe74f20b569b9ac8f8a7592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.205268 16002 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c251cfdbec94259be5afbb1b7dbaa38 with delete_type TABLET_DATA_TOMBSTONED (TS d1da7f11f0604f78934354a992bb0f55 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:16.207047 20462 tablet_bootstrap.cc:492] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:16.208704 20462 tablet_bootstrap.cc:654] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.213399 16308 tablet_copy_service.cc:342] P 5297e816f782430d9a01220f0471a882: Request end of tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-c78403038b614e6db2afb617328f6411 received from {username='slave'} at 127.6.196.195:53361
I20240502 14:15:16.213528 16308 tablet_copy_service.cc:433] P 5297e816f782430d9a01220f0471a882: ending tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-c78403038b614e6db2afb617328f6411 on tablet c78403038b614e6db2afb617328f6411 with peer 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:15:16.214696 16151 raft_consensus.cc:2932] T 1c251cfdbec94259be5afbb1b7dbaa38 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
W20240502 14:15:16.216949 13383 consensus_peers.cc:485] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3be42ac139741d19dd872ab551bebae. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:16.217972 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 1b6760f0772449088e93bc1eab5e2ab3 (DEFAULT_TABLE table=rebalancer_extra_table_48 [id=dbe0387864724576879d3ccdf6c2add8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:16.219271 16175 tablet_copy_service.cc:342] P c86f7440570f4796babfd44c3d7fd847: Request end of tablet copy session 5297e816f782430d9a01220f0471a882-db2bc203d47c4ca1b3329745a0ff7fe1 received from {username='slave'} at 127.6.196.199:32813
I20240502 14:15:16.219441 16175 tablet_copy_service.cc:433] P c86f7440570f4796babfd44c3d7fd847: ending tablet copy session 5297e816f782430d9a01220f0471a882-db2bc203d47c4ca1b3329745a0ff7fe1 on tablet db2bc203d47c4ca1b3329745a0ff7fe1 with peer 5297e816f782430d9a01220f0471a882
I20240502 14:15:16.220508 13187 raft_consensus.cc:2932] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:16.221904 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6760f0772449088e93bc1eab5e2ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.225507 20466 tablet_bootstrap.cc:492] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:16.227806 13038 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:16.228015 13038 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:16.231493 13038 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 89be9df9b5264b1b81fb5a22953f66d5
I20240502 14:15:16.232667 13168 tablet_service.cc:1514] Processing DeleteTablet for tablet 89be9df9b5264b1b81fb5a22953f66d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:16 UTC) from {username='slave'} at 127.0.0.1:37690
I20240502 14:15:16.232913 20417 tablet_replica.cc:330] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:15:16.233130 20417 raft_consensus.cc:2229] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:16.233350 20417 raft_consensus.cc:2258] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:16.234146 20417 ts_tablet_manager.cc:1875] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:16.235008 20466 tablet_bootstrap.cc:654] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.238509 16000 tablet_service.cc:1514] Processing DeleteTablet for tablet 89be9df9b5264b1b81fb5a22953f66d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:16 UTC) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:16.245867 20464 tablet_replica.cc:330] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:16.246415 20461 tablet_bootstrap.cc:492] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:16.248646 20464 raft_consensus.cc:2229] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:16.250264 16288 raft_consensus.cc:1203] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:16.251116 20464 raft_consensus.cc:2258] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:16.251948 20461 tablet_bootstrap.cc:654] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.252672 20464 ts_tablet_manager.cc:1875] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:16.262168 19413 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:16.264562 16399 tablet_service.cc:1514] Processing DeleteTablet for tablet 89be9df9b5264b1b81fb5a22953f66d5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:16 UTC) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:16.264827 20371 tablet_replica.cc:330] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:16.265043 20371 raft_consensus.cc:2229] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:16.265333 20371 raft_consensus.cc:2258] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:16.266326 20371 ts_tablet_manager.cc:1875] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:16.272907 20466 tablet_bootstrap.cc:492] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:16.273072 20466 ts_tablet_manager.cc:1367] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.004s
I20240502 14:15:16.273082 20464 ts_tablet_manager.cc:1888] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:16.273170 13038 catalog_manager.cc:5484] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 6 to 7, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:16.273875 20466 raft_consensus.cc:348] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:16.275761 20466 raft_consensus.cc:374] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.276701 20371 ts_tablet_manager.cc:1888] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:16.276859 20371 log.cc:1196] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/89be9df9b5264b1b81fb5a22953f66d5
I20240502 14:15:16.276842 20466 raft_consensus.cc:729] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.277042 20466 consensus_queue.cc:260] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:16.276180 13578 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:16.277559 20371 ts_tablet_manager.cc:1909] T 89be9df9b5264b1b81fb5a22953f66d5 P 8b47d4390f8343a493d6c00123785968: Deleting consensus metadata
I20240502 14:15:16.278553 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 89be9df9b5264b1b81fb5a22953f66d5 (table rebalancer_dynamic_table_new_name [id=5f63b67ca4304d1589b96cd94598d592]) successfully deleted
I20240502 14:15:16.280380 20417 ts_tablet_manager.cc:1888] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:16.280529 20417 log.cc:1196] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/89be9df9b5264b1b81fb5a22953f66d5
I20240502 14:15:16.280948 20417 ts_tablet_manager.cc:1909] T 89be9df9b5264b1b81fb5a22953f66d5 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting consensus metadata
I20240502 14:15:16.282017 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 89be9df9b5264b1b81fb5a22953f66d5 (table rebalancer_dynamic_table_new_name [id=5f63b67ca4304d1589b96cd94598d592]) successfully deleted
I20240502 14:15:16.283290 20462 tablet_bootstrap.cc:492] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:16.283543 20462 ts_tablet_manager.cc:1367] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.000s
I20240502 14:15:16.283783 20466 ts_tablet_manager.cc:1398] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240502 14:15:16.284729 20462 raft_consensus.cc:348] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:16.285043 20462 raft_consensus.cc:374] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.285266 20462 raft_consensus.cc:729] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.286792 16420 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:16.287390 20462 consensus_queue.cc:260] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:16.299190 20462 ts_tablet_manager.cc:1398] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:16.302132 20462 tablet_bootstrap.cc:492] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:16.303243 20100 raft_consensus.cc:482] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:16.303388 20100 raft_consensus.cc:504] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:16.303934 20100 leader_election.cc:288] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:16.304785 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274c6eb091a742089999cc39ad6dd76d" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:16.305392 13578 raft_consensus.cc:1203] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 14:15:16.305470 16353 leader_election.cc:341] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:16.307288 20462 tablet_bootstrap.cc:654] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.317433 20461 tablet_bootstrap.cc:492] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:16.317574 20461 ts_tablet_manager.cc:1367] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.000s
I20240502 14:15:16.318403 20461 raft_consensus.cc:348] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:16.319016 20461 raft_consensus.cc:374] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.319084 20461 raft_consensus.cc:729] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.319260 20461 consensus_queue.cc:260] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:16.321749 20461 ts_tablet_manager.cc:1398] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:16.323333 20463 tablet_bootstrap.cc:492] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:16.324911 20463 tablet_bootstrap.cc:654] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.328161 20470 tablet_bootstrap.cc:492] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:16.333461 20464 log.cc:1196] T 1c251cfdbec94259be5afbb1b7dbaa38 P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/1c251cfdbec94259be5afbb1b7dbaa38
I20240502 14:15:16.336458 16020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274c6eb091a742089999cc39ad6dd76d" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
W20240502 14:15:16.339428 16355 leader_election.cc:341] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:16.339551 16355 leader_election.cc:302] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8b47d4390f8343a493d6c00123785968; no voters: 88b77f6edc1c4a72b548bf47c05704f3, d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:16.340001 20311 raft_consensus.cc:2726] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:16.343151 20462 tablet_bootstrap.cc:492] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:16.343493 20462 ts_tablet_manager.cc:1367] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20240502 14:15:16.344580 20462 raft_consensus.cc:348] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:16.344705 20463 tablet_bootstrap.cc:492] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:16.344852 20463 ts_tablet_manager.cc:1367] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240502 14:15:16.344870 20462 raft_consensus.cc:374] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.344938 20462 raft_consensus.cc:729] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.345041 13038 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:16.345214 20462 consensus_queue.cc:260] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:16.345939 20463 raft_consensus.cc:348] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:16.346297 20463 raft_consensus.cc:374] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.347927 20462 ts_tablet_manager.cc:1398] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:16.346426 20463 raft_consensus.cc:729] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.349259 20470 tablet_bootstrap.cc:654] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.349236 20463 consensus_queue.cc:260] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:16.349886 20463 ts_tablet_manager.cc:1398] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:15:16.352506 20463 tablet_bootstrap.cc:492] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:16.357985 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet 1c251cfdbec94259be5afbb1b7dbaa38 (table rebalancer_extra_table_28 [id=fc31add041984706b1347b9ccdaecbaf]) successfully deleted
I20240502 14:15:16.358294 20464 tablet_replica.cc:330] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:16.358464 20464 raft_consensus.cc:2229] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:16.358723 20464 raft_consensus.cc:2258] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:16.363615 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 7159ec601b114965a12a44342f64c17b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4be65acc1ded45d9b1226ac1d7536ae3]), partition=
I20240502 14:15:16.363962 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7159ec601b114965a12a44342f64c17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.364184 16265 tablet_service.cc:1467] Processing CreateTablet for tablet 7159ec601b114965a12a44342f64c17b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4be65acc1ded45d9b1226ac1d7536ae3]), partition=
I20240502 14:15:16.364492 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7159ec601b114965a12a44342f64c17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.371239 20463 tablet_bootstrap.cc:654] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.370003 16002 tablet_service.cc:1467] Processing CreateTablet for tablet 7159ec601b114965a12a44342f64c17b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4be65acc1ded45d9b1226ac1d7536ae3]), partition=
I20240502 14:15:16.372687 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7159ec601b114965a12a44342f64c17b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.377550 20484 tablet_bootstrap.cc:492] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:16.378993 20484 tablet_bootstrap.cc:654] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.380344 20464 ts_tablet_manager.cc:1875] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:16.381314 20470 tablet_bootstrap.cc:492] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:16.381443 20470 ts_tablet_manager.cc:1367] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.002s
I20240502 14:15:16.382341 20470 raft_consensus.cc:348] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:16.382558 20470 raft_consensus.cc:374] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.382655 20470 raft_consensus.cc:729] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.382719 20465 tablet_bootstrap.cc:492] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:16.382894 20470 consensus_queue.cc:260] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:16.385633 13446 raft_consensus.cc:1203] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:16.389802 20470 ts_tablet_manager.cc:1398] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:15:16.391438 20470 tablet_bootstrap.cc:492] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:16.391521 13446 consensus_queue.cc:237] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:16.392284 20465 tablet_bootstrap.cc:654] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.394277 20470 tablet_bootstrap.cc:654] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.400914 16018 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:15:16.401098 13384 consensus_peers.cc:485] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9c5329f38f3343baaf43265990d86938. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:16.401422 20037 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:16.402714 13186 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:16.405537 20475 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:16.407260 20484 tablet_bootstrap.cc:492] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:16.407390 20484 ts_tablet_manager.cc:1367] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20240502 14:15:16.407920 20470 tablet_bootstrap.cc:492] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:16.408411 20484 raft_consensus.cc:348] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:16.408740 20463 tablet_bootstrap.cc:492] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:16.408771 20484 raft_consensus.cc:374] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.408874 20463 ts_tablet_manager.cc:1367] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.002s
I20240502 14:15:16.408988 20484 raft_consensus.cc:729] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.409298 20484 consensus_queue.cc:260] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:16.409476 20463 raft_consensus.cc:348] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:16.409653 20463 raft_consensus.cc:374] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.409734 20463 raft_consensus.cc:729] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.410001 20463 consensus_queue.cc:260] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:16.412544 20463 ts_tablet_manager.cc:1398] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:16.412809 20470 ts_tablet_manager.cc:1367] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20240502 14:15:16.415895 20484 ts_tablet_manager.cc:1398] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240502 14:15:16.421579 20470 raft_consensus.cc:348] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:16.421878 20470 raft_consensus.cc:374] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.421998 20470 raft_consensus.cc:729] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.422230 20470 consensus_queue.cc:260] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:16.424140 20470 ts_tablet_manager.cc:1398] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240502 14:15:16.438802 20464 ts_tablet_manager.cc:1888] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:16.438951 20464 log.cc:1196] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/89be9df9b5264b1b81fb5a22953f66d5
I20240502 14:15:16.439383 20464 ts_tablet_manager.cc:1909] T 89be9df9b5264b1b81fb5a22953f66d5 P d1da7f11f0604f78934354a992bb0f55: Deleting consensus metadata
I20240502 14:15:16.440227 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet 89be9df9b5264b1b81fb5a22953f66d5 (table rebalancer_dynamic_table_new_name [id=5f63b67ca4304d1589b96cd94598d592]) successfully deleted
I20240502 14:15:16.442852 20037 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:16.443466 13187 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:16.450577 13038 catalog_manager.cc:5484] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 6 to 7, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:16.452385 16023 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:16.467638 20465 tablet_bootstrap.cc:492] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:16.467811 20465 ts_tablet_manager.cc:1367] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.085s user 0.004s sys 0.002s
I20240502 14:15:16.468643 20465 raft_consensus.cc:348] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:16.468945 20465 raft_consensus.cc:374] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.469102 20465 raft_consensus.cc:729] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.469408 20465 consensus_queue.cc:260] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:16.473893 20465 ts_tablet_manager.cc:1398] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:15:16.481281 20465 tablet_bootstrap.cc:492] T 274c6eb091a742089999cc39ad6dd76d P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:16.483111 20465 tablet_bootstrap.cc:654] T 274c6eb091a742089999cc39ad6dd76d P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.497469 20475 raft_consensus.cc:482] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:16.497630 20475 raft_consensus.cc:504] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:16.498234 20465 tablet_bootstrap.cc:492] T 274c6eb091a742089999cc39ad6dd76d P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:16.498245 20475 leader_election.cc:288] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:16.498378 20465 ts_tablet_manager.cc:1367] T 274c6eb091a742089999cc39ad6dd76d P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20240502 14:15:16.499119 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274c6eb091a742089999cc39ad6dd76d" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:16.499228 20465 raft_consensus.cc:348] T 274c6eb091a742089999cc39ad6dd76d P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:16.499330 16419 raft_consensus.cc:2454] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:15:16.499434 20465 raft_consensus.cc:374] T 274c6eb091a742089999cc39ad6dd76d P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.499512 20465 raft_consensus.cc:729] T 274c6eb091a742089999cc39ad6dd76d P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.499992 13383 leader_election.cc:302] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:16.500422 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274c6eb091a742089999cc39ad6dd76d" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:16.500763 20465 consensus_queue.cc:260] T 274c6eb091a742089999cc39ad6dd76d P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:16.501020 16018 raft_consensus.cc:2454] T 274c6eb091a742089999cc39ad6dd76d P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:15:16.501885 20475 raft_consensus.cc:2781] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:16.501992 20475 raft_consensus.cc:482] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:16.502068 20475 raft_consensus.cc:3037] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.502435 20465 ts_tablet_manager.cc:1398] T 274c6eb091a742089999cc39ad6dd76d P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:16.502677 20465 tablet_bootstrap.cc:492] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:16.503926 20465 tablet_bootstrap.cc:654] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:16.504534 20475 raft_consensus.cc:504] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:16.508419 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274c6eb091a742089999cc39ad6dd76d" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:16.508598 16419 raft_consensus.cc:3037] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.510145 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274c6eb091a742089999cc39ad6dd76d" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:16.510396 16018 raft_consensus.cc:3037] T 274c6eb091a742089999cc39ad6dd76d P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.511948 16419 raft_consensus.cc:2454] T 274c6eb091a742089999cc39ad6dd76d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:16.512388 13383 leader_election.cc:302] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:16.512638 20037 raft_consensus.cc:2781] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:16.512740 20037 raft_consensus.cc:686] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:16.512974 20037 consensus_queue.cc:237] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:16.513136 16018 raft_consensus.cc:2454] T 274c6eb091a742089999cc39ad6dd76d P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:16.519326 13045 catalog_manager.cc:5484] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:16.520598 20475 leader_election.cc:288] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:16.528780 13319 consensus_queue.cc:237] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:16.529982 20465 tablet_bootstrap.cc:492] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:16.530146 20465 ts_tablet_manager.cc:1367] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.001s
I20240502 14:15:16.531337 20465 raft_consensus.cc:348] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:16.531621 20465 raft_consensus.cc:374] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:16.531834 20465 raft_consensus.cc:729] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:16.532132 20465 consensus_queue.cc:260] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:16.532912 20465 ts_tablet_manager.cc:1398] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20240502 14:15:16.537123 13253 consensus_peers.cc:485] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76c85c912ab74c32bfed536ea38d23f6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:16.538071 16419 raft_consensus.cc:1261] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:16.538549 16151 raft_consensus.cc:1261] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:16.538571 20387 consensus_queue.cc:1035] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:16.539551 20387 consensus_queue.cc:1035] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:16.542822 20087 raft_consensus.cc:2932] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:16.543339 16419 raft_consensus.cc:2932] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:16.543437 16151 raft_consensus.cc:2932] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:16.549144 13045 catalog_manager.cc:5484] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index -1 to 2, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:16.564546 13186 consensus_queue.cc:237] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:16.567559 16419 raft_consensus.cc:1261] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:16.567852 16287 raft_consensus.cc:1261] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:16.568289 13121 consensus_peers.cc:485] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 76b46f71180f4269bee53f206e9c2665. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:16.568295 20177 consensus_queue.cc:1035] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:16.568571 20512 consensus_queue.cc:1035] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:16.571269 20177 raft_consensus.cc:2932] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:16.572896 16419 raft_consensus.cc:2932] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:16.574605 16288 raft_consensus.cc:2932] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:16.576709 13038 catalog_manager.cc:5484] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:16.591838 13319 consensus_queue.cc:237] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:16.594033 16288 raft_consensus.cc:1261] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:16.594162 13578 raft_consensus.cc:1261] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:16.594502 13251 consensus_peers.cc:485] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c4f5c4167ac4c30af3aacb7f7cd099b. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:16.594525 20393 consensus_queue.cc:1035] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:16.594830 20087 consensus_queue.cc:1035] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:16.597322 20393 raft_consensus.cc:2932] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:16.597854 16288 raft_consensus.cc:2932] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:16.598165 13578 raft_consensus.cc:2932] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:16.603370 13038 catalog_manager.cc:5484] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:16.609819 20524 ts_tablet_manager.cc:902] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:16.620818 20524 tablet_copy_client.cc:308] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:15:16.621546 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet e3be42ac139741d19dd872ab551bebae from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:52085)
I20240502 14:15:16.621716 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet e3be42ac139741d19dd872ab551bebae from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:52085: session id = c86f7440570f4796babfd44c3d7fd847-e3be42ac139741d19dd872ab551bebae
I20240502 14:15:16.623886 13470 tablet_copy_source_session.cc:205] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:16.629848 20524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3be42ac139741d19dd872ab551bebae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:16.636207 20524 tablet_copy_client.cc:783] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:16.636441 20524 tablet_copy_client.cc:647] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:16.639005 20524 tablet_copy_client.cc:523] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:16.642634 20524 tablet_bootstrap.cc:492] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:16.671036 20468 raft_consensus.cc:482] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:16.671206 20468 raft_consensus.cc:504] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:16.671972 20468 leader_election.cc:288] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:16.672611 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e9c049ffe74f20b569b9ac8f8a7592" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:16.672821 16419 raft_consensus.cc:2454] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:16.673282 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e9c049ffe74f20b569b9ac8f8a7592" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:16.673494 16018 raft_consensus.cc:2454] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:16.674150 13515 leader_election.cc:302] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:16.675635 20468 raft_consensus.cc:2781] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:16.675719 20468 raft_consensus.cc:482] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:16.675781 20468 raft_consensus.cc:3037] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.677767 20468 raft_consensus.cc:504] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:16.678512 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e9c049ffe74f20b569b9ac8f8a7592" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:16.678681 16419 raft_consensus.cc:3037] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.679675 20512 raft_consensus.cc:482] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:16.679797 20512 raft_consensus.cc:504] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:16.680424 20512 leader_election.cc:288] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:16.681105 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6760f0772449088e93bc1eab5e2ab3" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:16.681336 13446 raft_consensus.cc:2454] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:15:16.681957 13120 leader_election.cc:302] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:16.681980 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6760f0772449088e93bc1eab5e2ab3" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:16.682201 16152 raft_consensus.cc:2454] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 0.
I20240502 14:15:16.681195 16419 raft_consensus.cc:2454] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:16.682592 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1e9c049ffe74f20b569b9ac8f8a7592" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:16.682773 16018 raft_consensus.cc:3037] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.684648 16018 raft_consensus.cc:2454] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:16.686161 20512 raft_consensus.cc:2781] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:16.686236 20512 raft_consensus.cc:482] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:16.686311 20512 raft_consensus.cc:3037] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.689129 20512 raft_consensus.cc:504] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:16.678227 20468 leader_election.cc:288] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:16.685940 13515 leader_election.cc:302] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:16.690987 13446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6760f0772449088e93bc1eab5e2ab3" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:16.691324 13446 raft_consensus.cc:3037] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.694490 20524 tablet_bootstrap.cc:492] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:16.695019 20524 tablet_bootstrap.cc:492] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:15:16.695295 20524 ts_tablet_manager.cc:1367] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.053s user 0.042s sys 0.004s
I20240502 14:15:16.696281 20468 raft_consensus.cc:2781] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:16.696177 20524 raft_consensus.cc:348] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:16.696568 20524 raft_consensus.cc:729] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:15:16.696751 20524 consensus_queue.cc:260] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:16.698549 20468 raft_consensus.cc:686] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:15:16.698871 20524 ts_tablet_manager.cc:1398] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:16.698769 20468 consensus_queue.cc:237] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:16.699668 13446 raft_consensus.cc:2454] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:15:16.700039 13120 leader_election.cc:302] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:16.700349 20177 raft_consensus.cc:2781] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:16.700469 20177 raft_consensus.cc:686] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:15:16.700721 20177 consensus_queue.cc:237] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:16.701912 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-e3be42ac139741d19dd872ab551bebae received from {username='slave'} at 127.6.196.198:52085
I20240502 14:15:16.702016 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-e3be42ac139741d19dd872ab551bebae on tablet e3be42ac139741d19dd872ab551bebae with peer c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:16.702576 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b6760f0772449088e93bc1eab5e2ab3" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:16.702757 16152 raft_consensus.cc:3037] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.705178 16152 raft_consensus.cc:2454] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:15:16.705654 20512 leader_election.cc:288] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 1 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:16.714542 13040 catalog_manager.cc:5484] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:16.724646 13040 catalog_manager.cc:5484] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:16.798655 19959 raft_consensus.cc:1050] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882: attempting to promote NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 to VOTER
I20240502 14:15:16.799402 19959 consensus_queue.cc:237] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:16.802770 13319 raft_consensus.cc:1261] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:16.803439 16018 raft_consensus.cc:1261] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:16.803386 20391 consensus_queue.cc:1035] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:16.804276 13578 raft_consensus.cc:1261] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:16.804440 20391 consensus_queue.cc:1035] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:16.804955 20391 consensus_queue.cc:1035] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:16.816570 20368 raft_consensus.cc:482] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:16.816752 20368 raft_consensus.cc:504] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:16.817767 20368 leader_election.cc:288] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:16.818313 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7159ec601b114965a12a44342f64c17b" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:16.818573 16023 raft_consensus.cc:2454] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:16.818578 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7159ec601b114965a12a44342f64c17b" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:16.818863 16152 raft_consensus.cc:2454] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:16.819036 16221 leader_election.cc:302] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:16.819350 20368 raft_consensus.cc:2781] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:16.819461 20368 raft_consensus.cc:482] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:16.819553 20368 raft_consensus.cc:3037] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.824925 20368 raft_consensus.cc:504] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:16.826136 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7159ec601b114965a12a44342f64c17b" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:16.826151 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7159ec601b114965a12a44342f64c17b" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:16.826324 16152 raft_consensus.cc:3037] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.826335 16023 raft_consensus.cc:3037] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:16.828929 16152 raft_consensus.cc:2454] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:16.829389 16220 leader_election.cc:302] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:16.829751 16023 raft_consensus.cc:2454] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:16.830289 20368 leader_election.cc:288] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:16.830772 20368 raft_consensus.cc:2932] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:16.868443 20582 raft_consensus.cc:2781] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:16.875347 19791 raft_consensus.cc:1050] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847: attempting to promote NON_VOTER 5297e816f782430d9a01220f0471a882 to VOTER
I20240502 14:15:16.876425 19791 consensus_queue.cc:237] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:16.881742 13446 raft_consensus.cc:1261] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:16.882520 20588 consensus_queue.cc:1035] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:16.882885 16023 raft_consensus.cc:1261] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:16.884748 16288 raft_consensus.cc:1261] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:16.885365 20588 consensus_queue.cc:1035] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:16.885624 18885 consensus_queue.cc:1035] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:15:16.906105 16220 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.196.198:42041, user_credentials={real_user=slave}} blocked reactor thread for 76753.1us
I20240502 14:15:16.918717 20582 raft_consensus.cc:686] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:16.921048 20582 consensus_queue.cc:237] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:16.927268 20475 consensus_queue.cc:1035] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:16.945273 13578 raft_consensus.cc:2932] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:16.954052 13040 catalog_manager.cc:5484] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:16.959848 20591 ts_tablet_manager.cc:902] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:16.961401 18885 raft_consensus.cc:2932] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:16.979702 16287 raft_consensus.cc:2932] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:16.988409 20586 raft_consensus.cc:1050] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882: attempting to promote NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 to VOTER
I20240502 14:15:16.989399 20586 consensus_queue.cc:237] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:16.992812 20474 consensus_queue.cc:1035] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.005357 13040 catalog_manager.cc:5484] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 2 to 3, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:17.074043 13446 raft_consensus.cc:2932] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:17.074649 16023 raft_consensus.cc:2932] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:17.084501 16151 consensus_queue.cc:237] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:17.090524 13045 catalog_manager.cc:5484] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:17.091351 13446 raft_consensus.cc:1261] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:17.103291 16151 raft_consensus.cc:1203] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:15:17.110512 20603 consensus_queue.cc:1035] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:17.113848 20603 raft_consensus.cc:2932] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:17.114220 20524 ts_tablet_manager.cc:902] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:17.115840 13446 raft_consensus.cc:2932] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:17.118338 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db2bc203d47c4ca1b3329745a0ff7fe1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:17.131042 13045 catalog_manager.cc:5484] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:17.147006 13319 raft_consensus.cc:2932] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:17.169638 20602 ts_tablet_manager.cc:902] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:17.171638 15999 tablet_service.cc:1514] Processing DeleteTablet for tablet db2bc203d47c4ca1b3329745a0ff7fe1 with delete_type TABLET_DATA_TOMBSTONED (TS d1da7f11f0604f78934354a992bb0f55 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41222
W20240502 14:15:17.161794 15955 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.193:36687, user_credentials={real_user=slave}} blocked reactor thread for 148600us
I20240502 14:15:17.174232 16419 raft_consensus.cc:1261] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:17.175611 20576 consensus_queue.cc:1035] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:17.176645 13446 raft_consensus.cc:1261] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:17.178395 16151 raft_consensus.cc:1261] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:17.179442 20582 consensus_queue.cc:1035] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:17.180845 20618 tablet_replica.cc:330] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:17.183373 20576 consensus_queue.cc:1035] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:17.188338 16288 raft_consensus.cc:1261] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:17.191150 20588 consensus_queue.cc:1035] T db2bc203d47c4ca1b3329745a0ff7fe1 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:17.198881 20602 tablet_copy_client.cc:308] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:15:17.199474 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 5c4f5c4167ac4c30af3aacb7f7cd099b from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:32917)
I20240502 14:15:17.199615 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 5c4f5c4167ac4c30af3aacb7f7cd099b from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:32917: session id = 20a967bad5504b1e9c6fb720ecdd1625-5c4f5c4167ac4c30af3aacb7f7cd099b
I20240502 14:15:17.202781 13339 tablet_copy_source_session.cc:205] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:17.203655 20619 ts_tablet_manager.cc:902] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:17.204360 16288 raft_consensus.cc:2932] T db2bc203d47c4ca1b3329745a0ff7fe1 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:17.204408 20582 raft_consensus.cc:2932] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:17.210800 13446 raft_consensus.cc:2932] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:17.212695 16152 raft_consensus.cc:2932] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:17.213956 13045 catalog_manager.cc:5484] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:17.215531 20618 raft_consensus.cc:2229] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:17.215783 20618 raft_consensus.cc:2258] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.215888 20177 consensus_queue.cc:1035] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.220868 16420 raft_consensus.cc:2932] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:17.225837 16288 consensus_queue.cc:237] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:17.228255 20602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c4f5c4167ac4c30af3aacb7f7cd099b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.230692 20591 tablet_copy_client.cc:308] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:15:17.231781 20618 ts_tablet_manager.cc:1875] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:17.232861 20177 consensus_queue.cc:1035] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.233538 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet 9c5329f38f3343baaf43265990d86938 from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:56181)
I20240502 14:15:17.233712 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet 9c5329f38f3343baaf43265990d86938 from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:56181: session id = 5297e816f782430d9a01220f0471a882-9c5329f38f3343baaf43265990d86938
I20240502 14:15:17.238008 13470 tablet_copy_source_session.cc:205] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:17.240859 16151 raft_consensus.cc:1261] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:17.241545 13446 raft_consensus.cc:1261] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:17.241811 20586 consensus_queue.cc:1035] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:17.242225 20586 consensus_queue.cc:1035] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:17.246086 20610 raft_consensus.cc:2932] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:17.250861 16151 raft_consensus.cc:2932] T c78403038b614e6db2afb617328f6411 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:17.257392 13446 raft_consensus.cc:2932] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:17.263026 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c78403038b614e6db2afb617328f6411 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:17.266913 20468 consensus_queue.cc:1035] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.266891 13039 catalog_manager.cc:5484] T c78403038b614e6db2afb617328f6411 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:17.268654 16018 raft_consensus.cc:2932] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:17.273576 16399 tablet_service.cc:1514] Processing DeleteTablet for tablet c78403038b614e6db2afb617328f6411 with delete_type TABLET_DATA_TOMBSTONED (TS 8b47d4390f8343a493d6c00123785968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:17.277648 20591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c5329f38f3343baaf43265990d86938. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.283852 20638 tablet_replica.cc:330] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:17.284121 20638 raft_consensus.cc:2229] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:17.284802 20638 raft_consensus.cc:2258] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.288960 16288 consensus_queue.cc:237] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:17.291568 13578 raft_consensus.cc:1261] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:17.292312 20581 consensus_queue.cc:1035] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:17.294500 20638 ts_tablet_manager.cc:1875] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:17.313910 13578 raft_consensus.cc:2932] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:17.312558 20586 raft_consensus.cc:2932] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:17.327263 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet caac1154570d4420ae8764ca18667f29 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:17.328857 20639 consensus_queue.cc:1035] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.330215 13039 catalog_manager.cc:5484] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 3 to 4, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:17.331879 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:17.335255 20591 tablet_copy_client.cc:783] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:17.335527 20591 tablet_copy_client.cc:647] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:17.335959 20581 consensus_queue.cc:1035] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.340379 20591 tablet_copy_client.cc:523] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:17.345532 13297 tablet_service.cc:1467] Processing CreateTablet for tablet ccee25274d364700921688f1dbd0c7b3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0023d4e1ffb7490fbf1c2d4cc2683818]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.345741 16018 raft_consensus.cc:1261] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:17.345747 15999 tablet_service.cc:1467] Processing CreateTablet for tablet f021ad6d95174c16a732234f0aa5ad76 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0023d4e1ffb7490fbf1c2d4cc2683818]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.345922 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccee25274d364700921688f1dbd0c7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.346213 15999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f021ad6d95174c16a732234f0aa5ad76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.348603 16265 tablet_service.cc:1467] Processing CreateTablet for tablet f021ad6d95174c16a732234f0aa5ad76 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0023d4e1ffb7490fbf1c2d4cc2683818]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.349011 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f021ad6d95174c16a732234f0aa5ad76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.350780 13166 tablet_service.cc:1467] Processing CreateTablet for tablet ccee25274d364700921688f1dbd0c7b3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0023d4e1ffb7490fbf1c2d4cc2683818]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.351171 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccee25274d364700921688f1dbd0c7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.352373 16135 tablet_service.cc:1467] Processing CreateTablet for tablet f021ad6d95174c16a732234f0aa5ad76 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0023d4e1ffb7490fbf1c2d4cc2683818]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.352674 16133 tablet_service.cc:1467] Processing CreateTablet for tablet ccee25274d364700921688f1dbd0c7b3 (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0023d4e1ffb7490fbf1c2d4cc2683818]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.352807 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f021ad6d95174c16a732234f0aa5ad76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.353040 16133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccee25274d364700921688f1dbd0c7b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.360476 20524 tablet_copy_client.cc:308] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:36687
I20240502 14:15:17.361291 13208 tablet_copy_service.cc:140] P 20a967bad5504b1e9c6fb720ecdd1625: Received BeginTabletCopySession request for tablet 76b46f71180f4269bee53f206e9c2665 from peer c86f7440570f4796babfd44c3d7fd847 ({username='slave'} at 127.6.196.198:46779)
I20240502 14:15:17.361461 13208 tablet_copy_service.cc:161] P 20a967bad5504b1e9c6fb720ecdd1625: Beginning new tablet copy session on tablet 76b46f71180f4269bee53f206e9c2665 from peer c86f7440570f4796babfd44c3d7fd847 at {username='slave'} at 127.6.196.198:46779: session id = c86f7440570f4796babfd44c3d7fd847-76b46f71180f4269bee53f206e9c2665
I20240502 14:15:17.365137 13208 tablet_copy_source_session.cc:205] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:17.366294 20638 ts_tablet_manager.cc:1888] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:17.367640 20638 log.cc:1196] T c78403038b614e6db2afb617328f6411 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/c78403038b614e6db2afb617328f6411
I20240502 14:15:17.368710 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet c78403038b614e6db2afb617328f6411 (table rebalancer_extra_table_26 [id=a2fc7359aefc4890a033bb4d687e45ff]) successfully deleted
I20240502 14:15:17.368690 16399 tablet_service.cc:1467] Processing CreateTablet for tablet ed5f3b784fa44b09974bc9caf17049af (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0023d4e1ffb7490fbf1c2d4cc2683818]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.369062 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5f3b784fa44b09974bc9caf17049af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.369640 20524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76b46f71180f4269bee53f206e9c2665. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.373723 20648 tablet_bootstrap.cc:492] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:17.375420 16002 tablet_service.cc:1467] Processing CreateTablet for tablet ed5f3b784fa44b09974bc9caf17049af (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0023d4e1ffb7490fbf1c2d4cc2683818]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.376214 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5f3b784fa44b09974bc9caf17049af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.380991 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet caac1154570d4420ae8764ca18667f29 with delete_type TABLET_DATA_TOMBSTONED (TS cd67493f6f0142f896010f2b8a39887d not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:57492
W20240502 14:15:17.382476 13515 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.200:43857, user_credentials={real_user=slave}} blocked reactor thread for 91430.6us
I20240502 14:15:17.394322 19959 consensus_queue.cc:1035] T caac1154570d4420ae8764ca18667f29 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:17.396986 20602 tablet_copy_client.cc:783] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:17.397322 20602 tablet_copy_client.cc:647] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:17.398454 13559 tablet_service.cc:1467] Processing CreateTablet for tablet ed5f3b784fa44b09974bc9caf17049af (DEFAULT_TABLE table=rebalancer_extra_table_50 [id=0023d4e1ffb7490fbf1c2d4cc2683818]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:17.398905 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5f3b784fa44b09974bc9caf17049af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.399910 16023 raft_consensus.cc:2932] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:17.404649 13038 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:17.405922 20602 tablet_copy_client.cc:523] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:17.417117 20648 tablet_bootstrap.cc:654] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.421928 20655 tablet_replica.cc:330] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:15:17.422343 20618 ts_tablet_manager.cc:1888] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:17.422607 20655 raft_consensus.cc:2229] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:17.422890 20655 raft_consensus.cc:2258] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.423049 20618 log.cc:1196] T db2bc203d47c4ca1b3329745a0ff7fe1 P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/db2bc203d47c4ca1b3329745a0ff7fe1
I20240502 14:15:17.424036 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet db2bc203d47c4ca1b3329745a0ff7fe1 (table rebalancer_extra_table_32 [id=6cab90f7f1c84cf2b82ee344af1b6617]) successfully deleted
I20240502 14:15:17.425552 20602 tablet_bootstrap.cc:492] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:17.426265 20655 ts_tablet_manager.cc:1875] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:17.432117 20653 tablet_bootstrap.cc:492] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:17.436488 20524 tablet_copy_client.cc:783] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:17.436722 20524 tablet_copy_client.cc:647] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:17.437423 20653 tablet_bootstrap.cc:654] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.438257 20651 tablet_bootstrap.cc:492] T ccee25274d364700921688f1dbd0c7b3 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:17.438977 20649 tablet_bootstrap.cc:492] T ccee25274d364700921688f1dbd0c7b3 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:17.439855 20651 tablet_bootstrap.cc:654] T ccee25274d364700921688f1dbd0c7b3 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.440086 20648 tablet_bootstrap.cc:492] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:17.440261 20648 ts_tablet_manager.cc:1367] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.000s
I20240502 14:15:17.441008 20648 raft_consensus.cc:348] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:17.441404 20649 tablet_bootstrap.cc:654] T ccee25274d364700921688f1dbd0c7b3 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.441301 20648 raft_consensus.cc:374] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.441640 20648 raft_consensus.cc:729] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.442032 20652 tablet_bootstrap.cc:492] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:17.442268 20648 consensus_queue.cc:260] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:17.442663 20656 tablet_bootstrap.cc:492] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:17.444597 20654 tablet_bootstrap.cc:492] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:17.446321 20648 ts_tablet_manager.cc:1398] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:15:17.446914 13045 catalog_manager.cc:5576] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857) does not have the latest schema for tablet 7159ec601b114965a12a44342f64c17b (table rebalancer_dynamic_table [id=4be65acc1ded45d9b1226ac1d7536ae3]). Expected version 1 got 0
I20240502 14:15:17.450629 20648 tablet_bootstrap.cc:492] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:17.450855 19959 consensus_queue.cc:1035] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.452708 20591 tablet_bootstrap.cc:492] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:17.439234 20524 tablet_copy_client.cc:523] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:17.456633 20648 tablet_bootstrap.cc:654] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.460244 20656 tablet_bootstrap.cc:654] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.462455 20649 tablet_bootstrap.cc:492] T ccee25274d364700921688f1dbd0c7b3 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:17.462589 20649 ts_tablet_manager.cc:1367] T ccee25274d364700921688f1dbd0c7b3 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.002s
I20240502 14:15:17.464049 20654 tablet_bootstrap.cc:654] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.467649 20652 tablet_bootstrap.cc:654] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.464123 20653 tablet_bootstrap.cc:492] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:17.469519 20653 ts_tablet_manager.cc:1367] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.002s
I20240502 14:15:17.471819 20653 raft_consensus.cc:348] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:17.470801 20649 raft_consensus.cc:348] T ccee25274d364700921688f1dbd0c7b3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:17.472314 20649 raft_consensus.cc:374] T ccee25274d364700921688f1dbd0c7b3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.472460 20649 raft_consensus.cc:729] T ccee25274d364700921688f1dbd0c7b3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.476692 20649 consensus_queue.cc:260] T ccee25274d364700921688f1dbd0c7b3 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:17.476951 20653 raft_consensus.cc:374] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.477092 20653 raft_consensus.cc:729] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.477810 20653 consensus_queue.cc:260] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:17.478253 20649 ts_tablet_manager.cc:1398] T ccee25274d364700921688f1dbd0c7b3 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.016s user 0.000s sys 0.003s
I20240502 14:15:17.483502 20653 ts_tablet_manager.cc:1398] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 14:15:17.479521 20668 tablet.cc:1722] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:17.484477 20668 tablet.cc:1620] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:17.486583 20654 tablet_bootstrap.cc:492] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:17.488272 20649 tablet_bootstrap.cc:492] T f021ad6d95174c16a732234f0aa5ad76 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:17.492369 20652 tablet_bootstrap.cc:492] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:17.492549 20652 ts_tablet_manager.cc:1367] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20240502 14:15:17.494127 20669 tablet.cc:1722] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:17.494855 20669 tablet.cc:1620] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:17.495538 20619 tablet_copy_client.cc:308] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:15:17.496363 20654 ts_tablet_manager.cc:1367] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.003s
I20240502 14:15:17.498162 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 76c85c912ab74c32bfed536ea38d23f6 from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:40779)
I20240502 14:15:17.498302 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 76c85c912ab74c32bfed536ea38d23f6 from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:40779: session id = d1da7f11f0604f78934354a992bb0f55-76c85c912ab74c32bfed536ea38d23f6
I20240502 14:15:17.499239 20649 tablet_bootstrap.cc:654] T f021ad6d95174c16a732234f0aa5ad76 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:17.500478 20524 tablet_bootstrap.cc:492] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:17.501183 13339 tablet_copy_source_session.cc:205] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:17.502014 20648 tablet_bootstrap.cc:492] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:17.502182 20648 ts_tablet_manager.cc:1367] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20240502 14:15:17.503116 20648 raft_consensus.cc:348] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:17.504683 20655 ts_tablet_manager.cc:1888] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:17.504841 20655 log.cc:1196] T caac1154570d4420ae8764ca18667f29 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/caac1154570d4420ae8764ca18667f29
I20240502 14:15:17.507100 20656 tablet_bootstrap.cc:492] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:17.507313 20656 ts_tablet_manager.cc:1367] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.065s user 0.002s sys 0.004s
I20240502 14:15:17.507905 20656 raft_consensus.cc:348] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:17.508116 20656 raft_consensus.cc:374] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.508227 20656 raft_consensus.cc:729] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.508502 20656 consensus_queue.cc:260] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:17.509387 20656 ts_tablet_manager.cc:1398] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:17.512418 20654 raft_consensus.cc:348] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:17.512686 20619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c85c912ab74c32bfed536ea38d23f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:17.515208 20648 raft_consensus.cc:374] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.515362 20648 raft_consensus.cc:729] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.515650 20581 raft_consensus.cc:482] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:17.515617 20648 consensus_queue.cc:260] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:17.515774 20581 raft_consensus.cc:504] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:17.516397 20648 ts_tablet_manager.cc:1398] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240502 14:15:17.516397 20581 leader_election.cc:288] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:17.516532 20654 raft_consensus.cc:374] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.516703 20654 raft_consensus.cc:729] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.506264 20652 raft_consensus.cc:348] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:17.517664 20652 raft_consensus.cc:374] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.517804 20652 raft_consensus.cc:729] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.518226 20654 consensus_queue.cc:260] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:17.516993 16019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f021ad6d95174c16a732234f0aa5ad76" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:17.518623 16019 raft_consensus.cc:2454] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:17.513612 20649 tablet_bootstrap.cc:492] T f021ad6d95174c16a732234f0aa5ad76 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:17.519309 20649 ts_tablet_manager.cc:1367] T f021ad6d95174c16a732234f0aa5ad76 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.004s
I20240502 14:15:17.519263 20652 consensus_queue.cc:260] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:17.519492 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f021ad6d95174c16a732234f0aa5ad76" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
W20240502 14:15:17.520328 16220 leader_election.cc:341] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:17.521157 13045 catalog_manager.cc:5576] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695) does not have the latest schema for tablet 7159ec601b114965a12a44342f64c17b (table rebalancer_dynamic_table [id=4be65acc1ded45d9b1226ac1d7536ae3]). Expected version 1 got 0
I20240502 14:15:17.522330 16221 leader_election.cc:302] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters: c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:17.522451 20649 raft_consensus.cc:348] T f021ad6d95174c16a732234f0aa5ad76 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:17.522586 20581 raft_consensus.cc:2781] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:17.522677 20581 raft_consensus.cc:482] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:17.522753 20581 raft_consensus.cc:3037] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.523752 20651 tablet_bootstrap.cc:492] T ccee25274d364700921688f1dbd0c7b3 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:17.523941 20651 ts_tablet_manager.cc:1367] T ccee25274d364700921688f1dbd0c7b3 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.086s user 0.002s sys 0.003s
I20240502 14:15:17.524559 20651 raft_consensus.cc:348] T ccee25274d364700921688f1dbd0c7b3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:17.525067 20581 raft_consensus.cc:504] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:17.525442 20651 raft_consensus.cc:374] T ccee25274d364700921688f1dbd0c7b3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.525439 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet caac1154570d4420ae8764ca18667f29 (table rebalancer_extra_table_34 [id=48aed9d8400b44738c091a427387612f]) successfully deleted
I20240502 14:15:17.526221 20581 leader_election.cc:288] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:17.527508 16019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f021ad6d95174c16a732234f0aa5ad76" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:17.527735 16019 raft_consensus.cc:3037] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.529098 20678 tablet.cc:1722] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:17.529727 20678 tablet.cc:1620] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:15:17.529959 13038 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=4be65acc1ded45d9b1226ac1d7536ae3] alter complete (version 1)
I20240502 14:15:17.530766 16019 raft_consensus.cc:2454] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:17.531213 16221 leader_election.cc:302] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:17.531500 20581 raft_consensus.cc:2781] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:17.531626 20581 raft_consensus.cc:686] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:17.531982 20581 consensus_queue.cc:237] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:17.534927 20652 ts_tablet_manager.cc:1398] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20240502 14:15:17.535086 20651 raft_consensus.cc:729] T ccee25274d364700921688f1dbd0c7b3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.535483 20654 ts_tablet_manager.cc:1398] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20240502 14:15:17.535941 20651 consensus_queue.cc:260] T ccee25274d364700921688f1dbd0c7b3 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:17.539598 13038 catalog_manager.cc:5484] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:17.526443 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f021ad6d95174c16a732234f0aa5ad76" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:17.545349 20649 raft_consensus.cc:374] T f021ad6d95174c16a732234f0aa5ad76 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:17.545430 20649 raft_consensus.cc:729] T f021ad6d95174c16a732234f0aa5ad76 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:17.545895 20649 consensus_queue.cc:260] T f021ad6d95174c16a732234f0aa5ad76 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:17.546136 16152 raft_consensus.cc:3037] T f021ad6d95174c16a732234f0aa5ad76 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.548434 16152 raft_consensus.cc:2454] T f021ad6d95174c16a732234f0aa5ad76 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:17.549454 20651 ts_tablet_manager.cc:1398] T ccee25274d364700921688f1dbd0c7b3 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.025s user 0.000s sys 0.002s
I20240502 14:15:17.558573 20602 tablet_bootstrap.cc:492] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:17.559219 20602 tablet_bootstrap.cc:492] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:15:17.559427 20602 ts_tablet_manager.cc:1367] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.134s user 0.004s sys 0.008s
I20240502 14:15:17.560101 20602 raft_consensus.cc:348] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:17.560307 20602 raft_consensus.cc:729] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:15:17.560496 20602 consensus_queue.cc:260] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:17.566371 20619 tablet_copy_client.cc:783] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:17.566665 20619 tablet_copy_client.cc:647] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:17.567553 20649 ts_tablet_manager.cc:1398] T f021ad6d95174c16a732234f0aa5ad76 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.047s user 0.001s sys 0.000s
I20240502 14:15:17.572782 20619 tablet_copy_client.cc:523] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:17.561048 20602 ts_tablet_manager.cc:1398] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:17.575976 20619 tablet_bootstrap.cc:492] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:17.580484 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-5c4f5c4167ac4c30af3aacb7f7cd099b received from {username='slave'} at 127.6.196.193:32917
I20240502 14:15:17.580621 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-5c4f5c4167ac4c30af3aacb7f7cd099b on tablet 5c4f5c4167ac4c30af3aacb7f7cd099b with peer 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:15:17.581038 20524 tablet_bootstrap.cc:492] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:17.584633 20524 tablet_bootstrap.cc:492] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847: Bootstrap complete.
I20240502 14:15:17.584872 20524 ts_tablet_manager.cc:1367] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.005s
I20240502 14:15:17.585553 20524 raft_consensus.cc:348] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:17.585781 20524 raft_consensus.cc:729] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: LEARNER
I20240502 14:15:17.585959 20524 consensus_queue.cc:260] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:17.591933 20524 ts_tablet_manager.cc:1398] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:15:17.596458 13208 tablet_copy_service.cc:342] P 20a967bad5504b1e9c6fb720ecdd1625: Request end of tablet copy session c86f7440570f4796babfd44c3d7fd847-76b46f71180f4269bee53f206e9c2665 received from {username='slave'} at 127.6.196.198:46779
I20240502 14:15:17.596577 13208 tablet_copy_service.cc:433] P 20a967bad5504b1e9c6fb720ecdd1625: ending tablet copy session c86f7440570f4796babfd44c3d7fd847-76b46f71180f4269bee53f206e9c2665 on tablet 76b46f71180f4269bee53f206e9c2665 with peer c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:17.618387 20619 tablet_bootstrap.cc:492] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:17.618876 20619 tablet_bootstrap.cc:492] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:15:17.619108 20619 ts_tablet_manager.cc:1367] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.043s user 0.016s sys 0.001s
I20240502 14:15:17.620532 20619 raft_consensus.cc:348] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:17.620806 20619 raft_consensus.cc:729] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:15:17.621024 20619 consensus_queue.cc:260] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:17.621593 20619 ts_tablet_manager.cc:1398] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:17.623307 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-76c85c912ab74c32bfed536ea38d23f6 received from {username='slave'} at 127.6.196.197:40779
I20240502 14:15:17.623399 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-76c85c912ab74c32bfed536ea38d23f6 on tablet 76c85c912ab74c32bfed536ea38d23f6 with peer d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:17.630710 20387 raft_consensus.cc:482] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:17.630854 20387 raft_consensus.cc:504] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:17.631476 20387 leader_election.cc:288] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:17.631896 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee25274d364700921688f1dbd0c7b3" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:17.632051 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee25274d364700921688f1dbd0c7b3" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:17.632129 16152 raft_consensus.cc:2454] T ccee25274d364700921688f1dbd0c7b3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:17.632239 13186 raft_consensus.cc:2454] T ccee25274d364700921688f1dbd0c7b3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:17.632567 13251 leader_election.cc:302] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:17.632938 20393 raft_consensus.cc:2781] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:17.632998 20393 raft_consensus.cc:482] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:17.633062 20393 raft_consensus.cc:3037] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.635284 20393 raft_consensus.cc:504] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:17.635803 20393 leader_election.cc:288] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:17.636767 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee25274d364700921688f1dbd0c7b3" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:17.636899 13186 raft_consensus.cc:3037] T ccee25274d364700921688f1dbd0c7b3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.636839 13038 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:17.639334 13186 raft_consensus.cc:2454] T ccee25274d364700921688f1dbd0c7b3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:17.644248 13251 leader_election.cc:302] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:17.645064 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccee25274d364700921688f1dbd0c7b3" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:17.645252 16152 raft_consensus.cc:3037] T ccee25274d364700921688f1dbd0c7b3 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.646168 20387 raft_consensus.cc:2781] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:17.647465 16152 raft_consensus.cc:2454] T ccee25274d364700921688f1dbd0c7b3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:17.650278 20669 tablet.cc:1722] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:17.650699 20669 tablet.cc:1620] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: MemRowSet was empty: no flush needed.
I20240502 14:15:17.658702 20678 tablet.cc:1722] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:17.659302 20678 tablet.cc:1620] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: MemRowSet was empty: no flush needed.
I20240502 14:15:17.659709 20387 raft_consensus.cc:686] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:17.660012 20387 consensus_queue.cc:237] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:17.668602 20668 tablet.cc:1722] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:17.669193 20668 tablet.cc:1620] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: MemRowSet was empty: no flush needed.
I20240502 14:15:17.670418 13038 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=4be65acc1ded45d9b1226ac1d7536ae3] alter complete (version 2)
I20240502 14:15:17.676998 20591 tablet_bootstrap.cc:492] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:17.684543 20643 raft_consensus.cc:482] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:17.684665 20643 raft_consensus.cc:504] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:17.685293 20643 leader_election.cc:288] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:17.685884 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5f3b784fa44b09974bc9caf17049af" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:17.686087 16420 raft_consensus.cc:2454] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:17.686344 13045 catalog_manager.cc:5484] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:17.686678 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5f3b784fa44b09974bc9caf17049af" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:17.686878 16023 raft_consensus.cc:2454] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:17.687222 13515 leader_election.cc:302] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:17.687757 20643 raft_consensus.cc:2781] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:17.687820 20643 raft_consensus.cc:482] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:17.687882 20643 raft_consensus.cc:3037] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.689779 20591 tablet_bootstrap.cc:492] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882: Bootstrap complete.
I20240502 14:15:17.690028 20591 ts_tablet_manager.cc:1367] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.237s user 0.047s sys 0.010s
I20240502 14:15:17.690055 20643 raft_consensus.cc:504] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:17.690668 20591 raft_consensus.cc:348] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:17.690865 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5f3b784fa44b09974bc9caf17049af" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:17.690946 20591 raft_consensus.cc:729] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: LEARNER
I20240502 14:15:17.691061 16420 raft_consensus.cc:3037] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.691150 20591 consensus_queue.cc:260] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:17.692724 20474 raft_consensus.cc:1050] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER c86f7440570f4796babfd44c3d7fd847 to VOTER
I20240502 14:15:17.693547 16420 raft_consensus.cc:2454] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:17.693598 20591 ts_tablet_manager.cc:1398] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:15:17.693499 20474 consensus_queue.cc:237] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:17.693866 13515 leader_election.cc:302] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:17.694084 20641 raft_consensus.cc:2781] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:17.694175 20641 raft_consensus.cc:686] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:15:17.694357 20641 consensus_queue.cc:237] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:17.695262 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session 5297e816f782430d9a01220f0471a882-9c5329f38f3343baaf43265990d86938 received from {username='slave'} at 127.6.196.199:56181
I20240502 14:15:17.695360 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session 5297e816f782430d9a01220f0471a882-9c5329f38f3343baaf43265990d86938 on tablet 9c5329f38f3343baaf43265990d86938 with peer 5297e816f782430d9a01220f0471a882
I20240502 14:15:17.695989 16419 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:17.696030 16152 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:17.696458 13579 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:17.696934 20474 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:17.697141 16023 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed5f3b784fa44b09974bc9caf17049af" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:17.697358 16023 raft_consensus.cc:3037] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:17.697319 20037 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:17.698323 20475 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:17.699954 16023 raft_consensus.cc:2454] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:17.701243 20643 leader_election.cc:288] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:17.702700 20474 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:17.705994 16154 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:17.705916 16420 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:17.709847 13045 catalog_manager.cc:5484] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 7 to 8, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:17.709857 13579 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:17.714742 13447 consensus_queue.cc:237] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:17.715088 13038 catalog_manager.cc:5484] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:17.723014 16419 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:17.725394 20475 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:17.727519 20474 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:17.727983 16419 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:17.730803 16152 raft_consensus.cc:1261] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:17.731251 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e3be42ac139741d19dd872ab551bebae with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:17.733783 20037 consensus_queue.cc:1035] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:17.734140 13038 catalog_manager.cc:5484] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:17.735689 16152 raft_consensus.cc:2932] T e3be42ac139741d19dd872ab551bebae P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:17.743824 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet e3be42ac139741d19dd872ab551bebae with delete_type TABLET_DATA_TOMBSTONED (TS 7eb4ecb2379b44c0954f2658c4a04350 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58526
I20240502 14:15:17.753917 16152 raft_consensus.cc:1203] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:17.763105 20704 tablet_replica.cc:330] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:15:17.763278 20704 raft_consensus.cc:2229] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:17.763446 20704 raft_consensus.cc:2258] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.764127 20704 ts_tablet_manager.cc:1875] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:17.771435 20704 ts_tablet_manager.cc:1888] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:15:17.771546 20704 log.cc:1196] T e3be42ac139741d19dd872ab551bebae P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/e3be42ac139741d19dd872ab551bebae
I20240502 14:15:17.776928 13045 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:17.777251 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet e3be42ac139741d19dd872ab551bebae (table rebalance_test_table_5 [id=c97a45b4ee8c48a38ebd0709da5770f7]) successfully deleted
I20240502 14:15:17.785835 20669 tablet.cc:1722] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:17.789114 20678 tablet.cc:1722] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:17.789809 20668 tablet.cc:1722] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:17.798461 13045 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=4be65acc1ded45d9b1226ac1d7536ae3] alter complete (version 3)
I20240502 14:15:17.847852 16288 raft_consensus.cc:1203] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:15:17.902631 13038 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:17.902848 13038 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:17.910955 13038 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet 7159ec601b114965a12a44342f64c17b
I20240502 14:15:17.912302 16133 tablet_service.cc:1514] Processing DeleteTablet for tablet 7159ec601b114965a12a44342f64c17b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:17 UTC) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:17.912783 16002 tablet_service.cc:1514] Processing DeleteTablet for tablet 7159ec601b114965a12a44342f64c17b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:17 UTC) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:17.912801 16265 tablet_service.cc:1514] Processing DeleteTablet for tablet 7159ec601b114965a12a44342f64c17b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:17 UTC) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:17.913069 20618 tablet_replica.cc:330] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:17.913383 20618 raft_consensus.cc:2229] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:17.913605 20618 raft_consensus.cc:2258] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.914216 13579 consensus_queue.cc:237] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:17.914512 20618 ts_tablet_manager.cc:1875] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:17.920749 20618 ts_tablet_manager.cc:1888] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:17.920953 20618 log.cc:1196] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/7159ec601b114965a12a44342f64c17b
I20240502 14:15:17.921448 20618 ts_tablet_manager.cc:1909] T 7159ec601b114965a12a44342f64c17b P d1da7f11f0604f78934354a992bb0f55: Deleting consensus metadata
I20240502 14:15:17.923339 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet 7159ec601b114965a12a44342f64c17b (table rebalancer_dynamic_table_new_name [id=4be65acc1ded45d9b1226ac1d7536ae3]) successfully deleted
I20240502 14:15:17.926574 20745 tablet_replica.cc:330] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: stopping tablet replica
W20240502 14:15:17.928664 13514 consensus_peers.cc:485] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 861f9171cb654ec889ddf824f2bb6e92. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:17.928926 20745 raft_consensus.cc:2229] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:17.929363 20745 raft_consensus.cc:2258] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.930536 16151 raft_consensus.cc:1261] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:17.931074 20641 consensus_queue.cc:1035] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:17.935189 20610 consensus_queue.cc:1035] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.939404 20745 ts_tablet_manager.cc:1875] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:17.940762 13187 raft_consensus.cc:1261] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:17.941321 20641 consensus_queue.cc:1035] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:17.961104 20641 raft_consensus.cc:2932] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:17.961617 13187 raft_consensus.cc:2932] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:17.969887 16152 raft_consensus.cc:2932] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:17.970561 20581 consensus_queue.cc:1035] T f021ad6d95174c16a732234f0aa5ad76 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:17.972849 13043 catalog_manager.cc:5484] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:17.981091 20745 ts_tablet_manager.cc:1888] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:17.981213 20745 log.cc:1196] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/7159ec601b114965a12a44342f64c17b
I20240502 14:15:17.981698 20745 ts_tablet_manager.cc:1909] T 7159ec601b114965a12a44342f64c17b P 5297e816f782430d9a01220f0471a882: Deleting consensus metadata
I20240502 14:15:17.982645 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 7159ec601b114965a12a44342f64c17b (table rebalancer_dynamic_table_new_name [id=4be65acc1ded45d9b1226ac1d7536ae3]) successfully deleted
I20240502 14:15:17.985287 20744 tablet_replica.cc:330] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:17.985498 20744 raft_consensus.cc:2229] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:17.985705 20744 raft_consensus.cc:2258] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:17.986732 20744 ts_tablet_manager.cc:1875] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:17.994197 13187 raft_consensus.cc:1203] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:17.997822 20744 ts_tablet_manager.cc:1888] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:17.999555 20744 log.cc:1196] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/7159ec601b114965a12a44342f64c17b
I20240502 14:15:18.000133 20744 ts_tablet_manager.cc:1909] T 7159ec601b114965a12a44342f64c17b P c86f7440570f4796babfd44c3d7fd847: Deleting consensus metadata
I20240502 14:15:18.001363 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 7159ec601b114965a12a44342f64c17b (table rebalancer_dynamic_table_new_name [id=4be65acc1ded45d9b1226ac1d7536ae3]) successfully deleted
I20240502 14:15:18.014276 13039 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:18.020465 16151 consensus_queue.cc:237] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:18.022887 16399 tablet_service.cc:1467] Processing CreateTablet for tablet a337c4f1550c42db8f8f46eaf600554f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=80ece6bedeed43b98d172a5546be930b]), partition=
I20240502 14:15:18.023237 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a337c4f1550c42db8f8f46eaf600554f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.024727 13579 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:15:18.025189 20588 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:18.025481 13559 tablet_service.cc:1467] Processing CreateTablet for tablet a337c4f1550c42db8f8f46eaf600554f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=80ece6bedeed43b98d172a5546be930b]), partition=
W20240502 14:15:18.025518 16088 consensus_peers.cc:485] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0e393a7736b14d5483973f9cb89058f8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:18.025563 13297 tablet_service.cc:1467] Processing CreateTablet for tablet a337c4f1550c42db8f8f46eaf600554f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=80ece6bedeed43b98d172a5546be930b]), partition=
I20240502 14:15:18.025817 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a337c4f1550c42db8f8f46eaf600554f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.025923 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a337c4f1550c42db8f8f46eaf600554f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.030579 20652 tablet_bootstrap.cc:492] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:18.031134 13319 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:15:18.031713 18885 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:18.031849 20652 tablet_bootstrap.cc:654] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.033406 20654 tablet_bootstrap.cc:492] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:18.033459 20652 tablet_bootstrap.cc:492] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:18.033556 20652 ts_tablet_manager.cc:1367] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:18.034265 20652 raft_consensus.cc:348] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:18.034443 20652 raft_consensus.cc:374] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.034492 20652 raft_consensus.cc:729] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.034662 20652 consensus_queue.cc:260] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:18.035004 20654 tablet_bootstrap.cc:654] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.035197 20652 ts_tablet_manager.cc:1398] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:18.046869 18885 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:18.050696 20758 tablet_bootstrap.cc:492] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:18.053037 20758 tablet_bootstrap.cc:654] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.054591 20758 tablet_bootstrap.cc:492] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:18.054273 13579 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:18.058779 13318 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:18.062752 20758 ts_tablet_manager.cc:1367] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240502 14:15:18.064142 20758 raft_consensus.cc:348] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:18.064455 20758 raft_consensus.cc:374] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.064560 20758 raft_consensus.cc:729] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.064812 20758 consensus_queue.cc:260] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:18.065433 20758 ts_tablet_manager.cc:1398] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:18.065814 13039 catalog_manager.cc:5484] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 7 to 8, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 2 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:18.073129 16019 raft_consensus.cc:1203] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:18.082607 20654 tablet_bootstrap.cc:492] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:18.082768 20654 ts_tablet_manager.cc:1367] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.049s user 0.008s sys 0.039s
I20240502 14:15:18.083560 20654 raft_consensus.cc:348] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:18.083847 20654 raft_consensus.cc:374] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.083969 20654 raft_consensus.cc:729] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.084218 20654 consensus_queue.cc:260] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:18.086055 20654 ts_tablet_manager.cc:1398] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:15:18.091403 20387 consensus_queue.cc:1035] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:18.098099 20393 consensus_queue.cc:1035] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:18.120762 20641 consensus_queue.cc:1035] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:18.129182 20642 consensus_queue.cc:1035] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:18.132661 16152 consensus_queue.cc:237] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:18.147478 16288 raft_consensus.cc:1261] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:18.147792 16090 consensus_peers.cc:485] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7b93f8449ef4e6fb0a7e315130da024. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:18.148169 18885 consensus_queue.cc:1035] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:18.148517 16023 raft_consensus.cc:1261] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:18.149045 18885 consensus_queue.cc:1035] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:18.151511 20603 raft_consensus.cc:2932] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:18.152359 16023 raft_consensus.cc:2932] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:18.158763 16288 raft_consensus.cc:2932] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:18.161543 13038 catalog_manager.cc:5484] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:18.181933 20177 raft_consensus.cc:1050] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625: attempting to promote NON_VOTER c86f7440570f4796babfd44c3d7fd847 to VOTER
I20240502 14:15:18.182750 20177 consensus_queue.cc:237] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:18.188333 16288 raft_consensus.cc:1261] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:18.188660 16152 raft_consensus.cc:1261] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEARNER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:18.188777 20512 consensus_queue.cc:1035] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:18.189291 20512 consensus_queue.cc:1035] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:18.203605 16420 raft_consensus.cc:1261] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:18.206049 20512 consensus_queue.cc:1035] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:18.207104 20177 raft_consensus.cc:2932] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:18.209950 16152 raft_consensus.cc:2932] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:18.210963 13319 consensus_queue.cc:237] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:18.211642 16287 raft_consensus.cc:2932] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:18.220252 13039 catalog_manager.cc:5484] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:18.221057 16420 raft_consensus.cc:2932] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:18.226584 13187 consensus_queue.cc:237] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } }
I20240502 14:15:18.228662 16287 raft_consensus.cc:1261] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:18.229393 20793 consensus_queue.cc:1035] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:18.229625 16287 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240502 14:15:18.229666 13252 consensus_peers.cc:485] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0854e9f4851e400f9af18181a71ae8f7. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:18.230505 13579 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:18.230751 16152 raft_consensus.cc:1261] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:18.231118 20387 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:18.231329 20177 consensus_queue.cc:1035] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:18.231863 20387 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:18.231854 20177 raft_consensus.cc:2932] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:18.233862 16288 raft_consensus.cc:2932] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:18.236449 16152 raft_consensus.cc:2932] T 76b46f71180f4269bee53f206e9c2665 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:18.236758 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 76b46f71180f4269bee53f206e9c2665 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:18.243351 20393 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:18.244745 13045 catalog_manager.cc:5484] T 76b46f71180f4269bee53f206e9c2665 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } }
I20240502 14:15:18.251647 16287 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:18.253336 16399 tablet_service.cc:1514] Processing DeleteTablet for tablet 76b46f71180f4269bee53f206e9c2665 with delete_type TABLET_DATA_TOMBSTONED (TS 8b47d4390f8343a493d6c00123785968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:18.254884 13040 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:18.255713 13579 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:18.256179 13038 catalog_manager.cc:5484] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:18.280506 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 430afdc079aa4c2b88be1e2a328a64bb (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b8a62418e89f462691470dc59226f75e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:18.281041 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430afdc079aa4c2b88be1e2a328a64bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.282286 16133 tablet_service.cc:1467] Processing CreateTablet for tablet 20f045499e874907bcde4318796580be (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b8a62418e89f462691470dc59226f75e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:18.282667 16133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f045499e874907bcde4318796580be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.285425 16135 tablet_service.cc:1467] Processing CreateTablet for tablet 4b7d267297334093aa0f65b121b16b61 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b8a62418e89f462691470dc59226f75e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:18.285787 16135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b7d267297334093aa0f65b121b16b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.286263 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 430afdc079aa4c2b88be1e2a328a64bb (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b8a62418e89f462691470dc59226f75e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:18.286613 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430afdc079aa4c2b88be1e2a328a64bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.289108 20804 tablet_bootstrap.cc:492] T 430afdc079aa4c2b88be1e2a328a64bb P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:18.289492 13298 tablet_service.cc:1467] Processing CreateTablet for tablet 20f045499e874907bcde4318796580be (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b8a62418e89f462691470dc59226f75e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:18.289801 13298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f045499e874907bcde4318796580be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.290913 20804 tablet_bootstrap.cc:654] T 430afdc079aa4c2b88be1e2a328a64bb P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.291190 13559 tablet_service.cc:1467] Processing CreateTablet for tablet 20f045499e874907bcde4318796580be (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b8a62418e89f462691470dc59226f75e]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:18.291530 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20f045499e874907bcde4318796580be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.292855 16002 tablet_service.cc:1467] Processing CreateTablet for tablet 4b7d267297334093aa0f65b121b16b61 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b8a62418e89f462691470dc59226f75e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:18.293345 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b7d267297334093aa0f65b121b16b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.281105 16401 tablet_service.cc:1467] Processing CreateTablet for tablet 430afdc079aa4c2b88be1e2a328a64bb (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b8a62418e89f462691470dc59226f75e]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:18.294715 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 430afdc079aa4c2b88be1e2a328a64bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.297326 13427 tablet_service.cc:1467] Processing CreateTablet for tablet 4b7d267297334093aa0f65b121b16b61 (DEFAULT_TABLE table=rebalancer_extra_table_52 [id=b8a62418e89f462691470dc59226f75e]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:18.297809 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b7d267297334093aa0f65b121b16b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.302502 20799 tablet_replica.cc:330] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:18.303340 20799 raft_consensus.cc:2229] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:18.304603 20799 raft_consensus.cc:2258] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:18.307657 20799 ts_tablet_manager.cc:1875] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:18.320489 20652 tablet_bootstrap.cc:492] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:18.322530 20652 tablet_bootstrap.cc:654] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.322655 20758 tablet_bootstrap.cc:492] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:18.323887 20758 tablet_bootstrap.cc:654] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.327910 20799 ts_tablet_manager.cc:1888] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:18.329380 20799 log.cc:1196] T 76b46f71180f4269bee53f206e9c2665 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/76b46f71180f4269bee53f206e9c2665
I20240502 14:15:18.330241 20654 tablet_bootstrap.cc:492] T 430afdc079aa4c2b88be1e2a328a64bb P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:18.330335 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 76b46f71180f4269bee53f206e9c2665 (table rebalancer_extra_table_34 [id=48aed9d8400b44738c091a427387612f]) successfully deleted
I20240502 14:15:18.331883 20654 tablet_bootstrap.cc:654] T 430afdc079aa4c2b88be1e2a328a64bb P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.336220 20652 tablet_bootstrap.cc:492] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:18.336447 20652 ts_tablet_manager.cc:1367] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20240502 14:15:18.337395 20652 raft_consensus.cc:348] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:18.337718 20652 raft_consensus.cc:374] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.337816 20652 raft_consensus.cc:729] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.338078 20652 consensus_queue.cc:260] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:18.347784 20652 ts_tablet_manager.cc:1398] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.011s user 0.000s sys 0.001s
I20240502 14:15:18.353188 20810 ts_tablet_manager.cc:902] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:18.354036 20652 tablet_bootstrap.cc:492] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:18.354910 20758 tablet_bootstrap.cc:492] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:18.355098 20758 ts_tablet_manager.cc:1367] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.005s
I20240502 14:15:18.355713 20652 tablet_bootstrap.cc:654] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.355755 20758 raft_consensus.cc:348] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:18.355976 20758 raft_consensus.cc:374] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.356055 20758 raft_consensus.cc:729] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.356386 20758 consensus_queue.cc:260] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:18.361138 20758 ts_tablet_manager.cc:1398] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240502 14:15:18.361536 20813 tablet_bootstrap.cc:492] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:18.361680 20806 tablet_bootstrap.cc:492] T 20f045499e874907bcde4318796580be P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:18.365039 20813 tablet_bootstrap.cc:654] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.375916 20806 tablet_bootstrap.cc:654] T 20f045499e874907bcde4318796580be P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.378703 20810 tablet_copy_client.cc:308] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:15:18.379283 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet 861f9171cb654ec889ddf824f2bb6e92 from peer 88b77f6edc1c4a72b548bf47c05704f3 ({username='slave'} at 127.6.196.195:49767)
I20240502 14:15:18.379413 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet 861f9171cb654ec889ddf824f2bb6e92 from peer 88b77f6edc1c4a72b548bf47c05704f3 at {username='slave'} at 127.6.196.195:49767: session id = 88b77f6edc1c4a72b548bf47c05704f3-861f9171cb654ec889ddf824f2bb6e92
I20240502 14:15:18.382252 13601 tablet_copy_source_session.cc:205] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:18.385577 20810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 861f9171cb654ec889ddf824f2bb6e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.393626 20813 tablet_bootstrap.cc:492] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:18.393743 20813 ts_tablet_manager.cc:1367] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.005s
I20240502 14:15:18.394345 20813 raft_consensus.cc:348] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:18.394590 20813 raft_consensus.cc:374] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.394712 20813 raft_consensus.cc:729] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.394989 20813 consensus_queue.cc:260] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:18.395349 20814 tablet_bootstrap.cc:492] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:18.398375 20814 tablet_bootstrap.cc:654] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.396761 20813 ts_tablet_manager.cc:1398] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:18.397114 20037 raft_consensus.cc:482] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.398778 20037 raft_consensus.cc:504] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:18.399329 20037 leader_election.cc:288] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:18.400378 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7d267297334093aa0f65b121b16b61" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
W20240502 14:15:18.400954 13383 leader_election.cc:341] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:18.401319 16019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7d267297334093aa0f65b121b16b61" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:18.401497 20652 tablet_bootstrap.cc:492] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d: No bootstrap required, opened a new log
I20240502 14:15:18.401646 20652 ts_tablet_manager.cc:1367] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.003s
W20240502 14:15:18.401898 13384 leader_election.cc:341] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:18.401993 13384 leader_election.cc:302] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:18.402421 20037 raft_consensus.cc:2726] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:18.402383 20652 raft_consensus.cc:348] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:18.402554 20652 raft_consensus.cc:374] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.402667 20652 raft_consensus.cc:729] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.402936 20652 consensus_queue.cc:260] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:18.407625 20810 tablet_copy_client.cc:783] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:18.407924 20810 tablet_copy_client.cc:647] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:18.408123 20652 ts_tablet_manager.cc:1398] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240502 14:15:18.410480 20810 tablet_copy_client.cc:523] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:18.415690 20654 tablet_bootstrap.cc:492] T 430afdc079aa4c2b88be1e2a328a64bb P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:18.415828 20654 ts_tablet_manager.cc:1367] T 430afdc079aa4c2b88be1e2a328a64bb P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.086s user 0.012s sys 0.020s
I20240502 14:15:18.416463 20654 raft_consensus.cc:348] T 430afdc079aa4c2b88be1e2a328a64bb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:18.416635 20654 raft_consensus.cc:374] T 430afdc079aa4c2b88be1e2a328a64bb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.416707 20654 raft_consensus.cc:729] T 430afdc079aa4c2b88be1e2a328a64bb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.416903 20654 consensus_queue.cc:260] T 430afdc079aa4c2b88be1e2a328a64bb P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:18.418370 20806 tablet_bootstrap.cc:492] T 20f045499e874907bcde4318796580be P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:18.418510 20806 ts_tablet_manager.cc:1367] T 20f045499e874907bcde4318796580be P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.005s
I20240502 14:15:18.418697 20814 tablet_bootstrap.cc:492] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:18.418949 20814 ts_tablet_manager.cc:1367] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240502 14:15:18.419301 20806 raft_consensus.cc:348] T 20f045499e874907bcde4318796580be P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:18.419514 20806 raft_consensus.cc:374] T 20f045499e874907bcde4318796580be P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.419612 20806 raft_consensus.cc:729] T 20f045499e874907bcde4318796580be P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.419921 20654 ts_tablet_manager.cc:1398] T 430afdc079aa4c2b88be1e2a328a64bb P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:15:18.419907 20806 consensus_queue.cc:260] T 20f045499e874907bcde4318796580be P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:18.424604 20474 raft_consensus.cc:1050] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER 5297e816f782430d9a01220f0471a882 to VOTER
I20240502 14:15:18.426863 20474 consensus_queue.cc:237] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:18.427904 20806 ts_tablet_manager.cc:1398] T 20f045499e874907bcde4318796580be P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20240502 14:15:18.428566 20814 raft_consensus.cc:348] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:18.430181 20814 raft_consensus.cc:374] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.430500 20814 raft_consensus.cc:729] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.430969 20814 consensus_queue.cc:260] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:18.437795 20806 tablet_bootstrap.cc:492] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847: Bootstrap starting.
I20240502 14:15:18.438325 20814 ts_tablet_manager.cc:1398] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240502 14:15:18.443055 20810 tablet_bootstrap.cc:492] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:18.443533 20806 tablet_bootstrap.cc:654] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:18.446923 20804 tablet_bootstrap.cc:492] T 430afdc079aa4c2b88be1e2a328a64bb P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:18.447079 20804 ts_tablet_manager.cc:1367] T 430afdc079aa4c2b88be1e2a328a64bb P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.158s user 0.031s sys 0.045s
I20240502 14:15:18.447930 20804 raft_consensus.cc:348] T 430afdc079aa4c2b88be1e2a328a64bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:18.448256 20804 raft_consensus.cc:374] T 430afdc079aa4c2b88be1e2a328a64bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.448356 20804 raft_consensus.cc:729] T 430afdc079aa4c2b88be1e2a328a64bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.448643 20804 consensus_queue.cc:260] T 430afdc079aa4c2b88be1e2a328a64bb P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:18.450134 16287 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882 [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:18.450739 20037 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:18.450870 20643 raft_consensus.cc:482] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.450994 20643 raft_consensus.cc:504] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:18.451627 20643 leader_election.cc:288] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:18.452222 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a337c4f1550c42db8f8f46eaf600554f" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:18.452503 16420 raft_consensus.cc:2454] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:18.453675 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a337c4f1550c42db8f8f46eaf600554f" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:18.453924 13319 raft_consensus.cc:2454] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:18.453155 13515 leader_election.cc:302] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:18.455116 20643 raft_consensus.cc:2781] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:18.455190 20804 ts_tablet_manager.cc:1398] T 430afdc079aa4c2b88be1e2a328a64bb P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240502 14:15:18.455219 20643 raft_consensus.cc:482] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:18.455304 20643 raft_consensus.cc:3037] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.455971 16019 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:18.456468 20037 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:18.458067 20643 raft_consensus.cc:504] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:18.459055 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a337c4f1550c42db8f8f46eaf600554f" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:18.459300 16420 raft_consensus.cc:3037] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.462067 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a337c4f1550c42db8f8f46eaf600554f" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:18.462253 13319 raft_consensus.cc:3037] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.463054 20643 leader_election.cc:288] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:18.464792 13186 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:18.465121 13319 raft_consensus.cc:2454] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:18.466715 20691 raft_consensus.cc:1050] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 to VOTER
I20240502 14:15:18.467351 16420 raft_consensus.cc:2454] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:18.467542 20691 consensus_queue.cc:237] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:18.467917 13516 leader_election.cc:302] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:18.467726 20393 raft_consensus.cc:482] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.468426 20393 raft_consensus.cc:504] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:18.469084 20393 leader_election.cc:288] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:18.469923 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f045499e874907bcde4318796580be" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:18.470208 16152 raft_consensus.cc:2454] T 20f045499e874907bcde4318796580be P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:18.470736 13252 leader_election.cc:302] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:18.471019 20387 raft_consensus.cc:2781] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:18.471088 20387 raft_consensus.cc:482] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:18.471150 20387 raft_consensus.cc:3037] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.473656 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f045499e874907bcde4318796580be" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:18.473815 13579 raft_consensus.cc:2454] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:18.474036 20642 raft_consensus.cc:2781] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:18.474131 20642 raft_consensus.cc:686] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:15:18.474331 20642 consensus_queue.cc:237] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:18.474403 20037 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:18.475782 20641 raft_consensus.cc:482] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.475912 20641 raft_consensus.cc:504] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:18.476397 20641 leader_election.cc:288] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:18.479135 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f045499e874907bcde4318796580be" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
I20240502 14:15:18.479360 16152 raft_consensus.cc:2454] T 20f045499e874907bcde4318796580be P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 0.
I20240502 14:15:18.479763 13515 leader_election.cc:302] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, c86f7440570f4796babfd44c3d7fd847; no voters:
I20240502 14:15:18.480070 20641 raft_consensus.cc:2781] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:18.480151 20641 raft_consensus.cc:482] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:18.480237 20641 raft_consensus.cc:3037] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.480643 20387 raft_consensus.cc:504] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:18.482345 13579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f045499e874907bcde4318796580be" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:18.482724 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f045499e874907bcde4318796580be" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:18.482898 16152 raft_consensus.cc:3037] T 20f045499e874907bcde4318796580be P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.483139 20641 raft_consensus.cc:504] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:18.483395 13579 raft_consensus.cc:2379] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate cd67493f6f0142f896010f2b8a39887d in current term 1: Already voted for candidate 7eb4ecb2379b44c0954f2658c4a04350 in this term.
I20240502 14:15:18.484071 20641 leader_election.cc:288] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:18.486114 16151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f045499e874907bcde4318796580be" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:18.487028 16152 raft_consensus.cc:2454] T 20f045499e874907bcde4318796580be P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:18.487504 13252 leader_election.cc:302] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d; no voters: 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:15:18.490495 20387 leader_election.cc:288] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:18.490840 20387 raft_consensus.cc:2781] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:18.490962 20387 raft_consensus.cc:686] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:18.491184 20387 consensus_queue.cc:237] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:18.493146 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f045499e874907bcde4318796580be" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:18.509609 13186 raft_consensus.cc:1261] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:18.512286 20837 consensus_queue.cc:1035] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:18.512980 13579 raft_consensus.cc:1261] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:18.513514 20387 consensus_queue.cc:1035] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:18.513442 20037 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:18.517421 20609 raft_consensus.cc:482] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.517603 20609 raft_consensus.cc:504] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:18.518349 20609 leader_election.cc:288] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:18.519326 13040 catalog_manager.cc:5484] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:18.519626 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7d267297334093aa0f65b121b16b61" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:18.519815 13445 raft_consensus.cc:2454] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:18.520989 15955 leader_election.cc:302] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:18.521770 20789 raft_consensus.cc:2781] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:18.521872 20789 raft_consensus.cc:482] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:18.521982 20789 raft_consensus.cc:3037] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.524840 20789 raft_consensus.cc:504] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:18.527521 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7d267297334093aa0f65b121b16b61" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847" is_pre_election: true
W20240502 14:15:18.528000 15956 leader_election.cc:341] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:18.528621 13186 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:18.530644 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7d267297334093aa0f65b121b16b61" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:18.530820 13445 raft_consensus.cc:3037] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.531562 13040 catalog_manager.cc:5484] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 7 to 8, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:18.533464 13445 raft_consensus.cc:2454] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:18.533733 20789 leader_election.cc:288] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:18.534866 15955 leader_election.cc:302] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:18.535634 16151 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7d267297334093aa0f65b121b16b61" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
W20240502 14:15:18.536114 15956 leader_election.cc:341] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:18.539698 20789 raft_consensus.cc:2781] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:18.539968 20789 raft_consensus.cc:686] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:18.540249 20789 consensus_queue.cc:237] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
W20240502 14:15:18.548414 13253 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.197:43695, user_credentials={real_user=slave}} blocked reactor thread for 70552.9us
I20240502 14:15:18.554608 16019 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:18.558019 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20f045499e874907bcde4318796580be" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:18.558774 13516 leader_election.cc:302] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:18.558876 16283 raft_consensus.cc:1261] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:18.559216 20643 raft_consensus.cc:2726] T 20f045499e874907bcde4318796580be P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240502 14:15:18.567508 20691 consensus_queue.cc:1035] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:15:18.575004 13252 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.196.198:42041, user_credentials={real_user=slave}} blocked reactor thread for 87515.7us
I20240502 14:15:18.576802 13045 catalog_manager.cc:5484] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:18.579322 20691 raft_consensus.cc:2932] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:18.582264 13040 catalog_manager.cc:5484] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:18.583334 20824 ts_tablet_manager.cc:902] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:18.584246 13581 raft_consensus.cc:2932] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:18.595291 13187 raft_consensus.cc:2932] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:18.599556 13040 catalog_manager.cc:5484] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:18.600389 20806 tablet_bootstrap.cc:492] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847: No bootstrap required, opened a new log
I20240502 14:15:18.601470 20810 tablet_bootstrap.cc:492] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:18.601843 20810 tablet_bootstrap.cc:492] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap complete.
I20240502 14:15:18.602097 20810 ts_tablet_manager.cc:1367] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.159s user 0.004s sys 0.002s
I20240502 14:15:18.602748 20810 raft_consensus.cc:348] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:18.602985 20810 raft_consensus.cc:729] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: LEARNER
I20240502 14:15:18.603163 20810 consensus_queue.cc:260] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:18.603718 20810 ts_tablet_manager.cc:1398] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:18.603974 13581 consensus_queue.cc:237] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:18.606025 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-861f9171cb654ec889ddf824f2bb6e92 received from {username='slave'} at 127.6.196.195:49767
I20240502 14:15:18.606112 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-861f9171cb654ec889ddf824f2bb6e92 on tablet 861f9171cb654ec889ddf824f2bb6e92 with peer 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:15:18.605959 16288 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 5297e816f782430d9a01220f0471a882 (127.6.196.199) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:18.610402 13445 consensus_queue.cc:237] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:18.615634 16419 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:18.616201 16023 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:18.616443 20641 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:18.616626 20475 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:18.617523 13447 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:18.617995 20643 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:18.621467 20824 tablet_copy_client.cc:308] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.198:42041
I20240502 14:15:18.621747 20643 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:18.624320 16285 raft_consensus.cc:1261] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:18.624864 20475 consensus_queue.cc:1035] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:18.625866 16419 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:18.626868 20475 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:18.629989 16285 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:18.632309 13040 catalog_manager.cc:5484] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:18.638710 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9c5329f38f3343baaf43265990d86938 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:18.643766 13040 catalog_manager.cc:5484] T 9c5329f38f3343baaf43265990d86938 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 8 to 9, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:18.646880 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet 9c5329f38f3343baaf43265990d86938 with delete_type TABLET_DATA_TOMBSTONED (TS 20a967bad5504b1e9c6fb720ecdd1625 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:37690
W20240502 14:15:18.649418 13516 consensus_peers.cc:485] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): Couldn't send request to peer d1da7f11f0604f78934354a992bb0f55. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 27857198d4ec430ebee897c90141c58d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:18.622218 16175 tablet_copy_service.cc:140] P c86f7440570f4796babfd44c3d7fd847: Received BeginTabletCopySession request for tablet 0e393a7736b14d5483973f9cb89058f8 from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:41633)
I20240502 14:15:18.649638 20806 ts_tablet_manager.cc:1367] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847: Time spent bootstrapping tablet: real 0.212s user 0.000s sys 0.007s
I20240502 14:15:18.649642 16175 tablet_copy_service.cc:161] P c86f7440570f4796babfd44c3d7fd847: Beginning new tablet copy session on tablet 0e393a7736b14d5483973f9cb89058f8 from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:41633: session id = 20a967bad5504b1e9c6fb720ecdd1625-0e393a7736b14d5483973f9cb89058f8
I20240502 14:15:18.650441 20806 raft_consensus.cc:348] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:18.650672 20806 raft_consensus.cc:374] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:18.650759 20806 raft_consensus.cc:729] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86f7440570f4796babfd44c3d7fd847, State: Initialized, Role: FOLLOWER
I20240502 14:15:18.651656 13319 consensus_queue.cc:237] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:18.651556 13445 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } } }
I20240502 14:15:18.652889 16175 tablet_copy_source_session.cc:205] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:18.653235 16287 raft_consensus.cc:2932] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:18.654436 13185 raft_consensus.cc:1261] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:18.655438 20835 consensus_queue.cc:1035] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:18.656112 13581 raft_consensus.cc:1261] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:18.658903 20835 consensus_queue.cc:1035] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:18.658871 20806 consensus_queue.cc:260] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:18.659533 20835 raft_consensus.cc:2932] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:18.662253 20824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e393a7736b14d5483973f9cb89058f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.664499 20806 ts_tablet_manager.cc:1398] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240502 14:15:18.665861 16023 raft_consensus.cc:2932] T 9c5329f38f3343baaf43265990d86938 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } }
I20240502 14:15:18.669692 13185 raft_consensus.cc:2932] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:18.676223 13043 catalog_manager.cc:5484] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:18.678175 20850 tablet_replica.cc:330] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:15:18.678376 20850 raft_consensus.cc:2229] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:18.678704 20850 raft_consensus.cc:2258] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:18.678683 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5c4f5c4167ac4c30af3aacb7f7cd099b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:18.681073 20824 tablet_copy_client.cc:783] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:18.681324 20824 tablet_copy_client.cc:647] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:18.682552 20850 ts_tablet_manager.cc:1875] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:18.684028 20824 tablet_copy_client.cc:523] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:18.687763 13581 raft_consensus.cc:2932] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:18.688194 20824 tablet_bootstrap.cc:492] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:18.696625 16265 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c4f5c4167ac4c30af3aacb7f7cd099b with delete_type TABLET_DATA_TOMBSTONED (TS 5297e816f782430d9a01220f0471a882 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42272
I20240502 14:15:18.703696 20837 raft_consensus.cc:1050] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER d1da7f11f0604f78934354a992bb0f55 to VOTER
I20240502 14:15:18.704450 20837 consensus_queue.cc:237] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:18.720005 13445 raft_consensus.cc:1203] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:18.722370 20852 tablet_replica.cc:330] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882: stopping tablet replica
I20240502 14:15:18.722579 20852 raft_consensus.cc:2229] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:18.722796 20852 raft_consensus.cc:2258] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:18.724094 16019 raft_consensus.cc:1261] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:18.724238 20852 ts_tablet_manager.cc:1875] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:18.724617 20834 consensus_queue.cc:1035] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:18.728570 20850 ts_tablet_manager.cc:1888] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:15:18.728788 20850 log.cc:1196] T 9c5329f38f3343baaf43265990d86938 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/9c5329f38f3343baaf43265990d86938
I20240502 14:15:18.729627 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet 9c5329f38f3343baaf43265990d86938 (table rebalance_test_table_3 [id=8fa288258a0f4231be6de034088616ab]) successfully deleted
I20240502 14:15:18.730289 20855 ts_tablet_manager.cc:902] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d: Initiating tablet copy from peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:18.730508 16152 raft_consensus.cc:1261] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:18.733239 20834 consensus_queue.cc:1035] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:18.736065 13040 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240502 14:15:18.736335 20834 raft_consensus.cc:482] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:18.736469 20834 raft_consensus.cc:504] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:18.737990 16420 raft_consensus.cc:1261] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:18.739291 20834 leader_election.cc:288] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:18.739321 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430afdc079aa4c2b88be1e2a328a64bb" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:18.739526 13185 raft_consensus.cc:2454] T 430afdc079aa4c2b88be1e2a328a64bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:18.739490 20834 consensus_queue.cc:1035] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:18.740056 13251 leader_election.cc:302] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:18.742259 20837 raft_consensus.cc:2781] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:18.742394 20837 raft_consensus.cc:482] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:18.742529 20837 raft_consensus.cc:3037] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.745275 20834 raft_consensus.cc:2932] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:18.745908 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430afdc079aa4c2b88be1e2a328a64bb" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:18.745852 16023 raft_consensus.cc:2932] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:18.746177 16420 raft_consensus.cc:2454] T 430afdc079aa4c2b88be1e2a328a64bb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 0.
I20240502 14:15:18.754875 13045 catalog_manager.cc:5484] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 2 to 3, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:18.756562 20837 raft_consensus.cc:504] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:18.758790 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430afdc079aa4c2b88be1e2a328a64bb" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:18.759004 13185 raft_consensus.cc:3037] T 430afdc079aa4c2b88be1e2a328a64bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.761770 13185 raft_consensus.cc:2454] T 430afdc079aa4c2b88be1e2a328a64bb P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:18.761534 16151 raft_consensus.cc:2932] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:18.763235 13251 leader_election.cc:302] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:18.763839 20387 raft_consensus.cc:2781] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:18.763978 20387 raft_consensus.cc:686] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:18.764236 20387 consensus_queue.cc:237] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:18.773365 20867 ts_tablet_manager.cc:902] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:18.776852 16420 raft_consensus.cc:1261] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:18.777433 20641 consensus_queue.cc:1035] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:18.782014 20837 leader_election.cc:288] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:18.782868 16418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "430afdc079aa4c2b88be1e2a328a64bb" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:18.783161 16418 raft_consensus.cc:3037] T 430afdc079aa4c2b88be1e2a328a64bb P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:18.785033 13319 raft_consensus.cc:1261] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:18.785570 20641 consensus_queue.cc:1035] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:18.794672 13040 catalog_manager.cc:5484] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 0 to 1, leader changed from <none> to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:18.801788 16418 raft_consensus.cc:2454] T 430afdc079aa4c2b88be1e2a328a64bb P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:18.806619 20852 ts_tablet_manager.cc:1888] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:18.811326 20852 log.cc:1196] T 5c4f5c4167ac4c30af3aacb7f7cd099b P 5297e816f782430d9a01220f0471a882: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/wals/5c4f5c4167ac4c30af3aacb7f7cd099b
I20240502 14:15:18.812340 13031 catalog_manager.cc:4830] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): tablet 5c4f5c4167ac4c30af3aacb7f7cd099b (table rebalancer_extra_table_38 [id=82dbfe603cd54bde8f4e20b802fcca34]) successfully deleted
I20240502 14:15:18.818974 16421 raft_consensus.cc:2932] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:18.824591 20855 tablet_copy_client.cc:308] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.198:42041
I20240502 14:15:18.825251 16175 tablet_copy_service.cc:140] P c86f7440570f4796babfd44c3d7fd847: Received BeginTabletCopySession request for tablet e7b93f8449ef4e6fb0a7e315130da024 from peer cd67493f6f0142f896010f2b8a39887d ({username='slave'} at 127.6.196.194:59023)
I20240502 14:15:18.825429 16175 tablet_copy_service.cc:161] P c86f7440570f4796babfd44c3d7fd847: Beginning new tablet copy session on tablet e7b93f8449ef4e6fb0a7e315130da024 from peer cd67493f6f0142f896010f2b8a39887d at {username='slave'} at 127.6.196.194:59023: session id = cd67493f6f0142f896010f2b8a39887d-e7b93f8449ef4e6fb0a7e315130da024
I20240502 14:15:18.828794 16175 tablet_copy_source_session.cc:205] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:18.830315 20855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7b93f8449ef4e6fb0a7e315130da024. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.834937 13040 catalog_manager.cc:5576] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193) does not have the latest schema for tablet a337c4f1550c42db8f8f46eaf600554f (table rebalancer_dynamic_table [id=80ece6bedeed43b98d172a5546be930b]). Expected version 1 got 0
I20240502 14:15:18.837344 20877 tablet.cc:1722] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:18.837838 20877 tablet.cc:1620] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: MemRowSet was empty: no flush needed.
I20240502 14:15:18.840255 20878 tablet.cc:1722] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:18.846568 13040 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=80ece6bedeed43b98d172a5546be930b] alter complete (version 1)
I20240502 14:15:18.848040 13045 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet a337c4f1550c42db8f8f46eaf600554f (table rebalancer_dynamic_table [id=80ece6bedeed43b98d172a5546be930b]). Expected version 1 got 0
I20240502 14:15:18.872031 20878 tablet.cc:1620] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: MemRowSet was empty: no flush needed.
I20240502 14:15:18.874609 20855 tablet_copy_client.cc:783] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:18.875545 20855 tablet_copy_client.cc:647] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:18.880506 20855 tablet_copy_client.cc:523] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:18.881146 16019 consensus_queue.cc:237] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:18.894726 20855 tablet_bootstrap.cc:492] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:18.912714 13187 raft_consensus.cc:1261] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:18.913542 20609 consensus_queue.cc:1035] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:18.914561 13447 raft_consensus.cc:1261] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:18.914762 15956 consensus_peers.cc:485] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fae5b25ec9694cea87381d27f4d46384. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:18.915210 20609 consensus_queue.cc:1035] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:18.920032 20609 raft_consensus.cc:2932] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:18.922474 13185 raft_consensus.cc:2932] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:18.923132 20867 tablet_copy_client.cc:308] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:15:18.923858 13040 catalog_manager.cc:5576] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857) does not have the latest schema for tablet a337c4f1550c42db8f8f46eaf600554f (table rebalancer_dynamic_table [id=80ece6bedeed43b98d172a5546be930b]). Expected version 1 got 0
I20240502 14:15:18.926748 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 0854e9f4851e400f9af18181a71ae8f7 from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:56671)
I20240502 14:15:18.926867 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 0854e9f4851e400f9af18181a71ae8f7 from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:56671: session id = 8b47d4390f8343a493d6c00123785968-0854e9f4851e400f9af18181a71ae8f7
I20240502 14:15:18.930496 13040 catalog_manager.cc:5484] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:18.933969 13339 tablet_copy_source_session.cc:205] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:18.936839 13447 raft_consensus.cc:2932] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:18.939025 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0854e9f4851e400f9af18181a71ae8f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:18.919600 20880 tablet.cc:1722] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240502 14:15:18.942656 20880 tablet.cc:1620] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:18.957090 20867 tablet_copy_client.cc:783] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:18.957774 13045 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240502 14:15:18.965415 20867 tablet_copy_client.cc:647] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:18.967013 20877 tablet.cc:1722] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:18.967375 20877 tablet.cc:1620] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: MemRowSet was empty: no flush needed.
I20240502 14:15:18.968005 20880 tablet.cc:1722] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:18.969107 20880 tablet.cc:1620] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: MemRowSet was empty: no flush needed.
I20240502 14:15:18.972627 20855 tablet_bootstrap.cc:492] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:18.973480 20867 tablet_copy_client.cc:523] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:18.974910 13447 consensus_queue.cc:237] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:18.976215 20878 tablet.cc:1722] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240502 14:15:18.976425 20867 tablet_bootstrap.cc:492] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:18.976709 20878 tablet.cc:1620] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: MemRowSet was empty: no flush needed.
I20240502 14:15:18.981477 20855 tablet_bootstrap.cc:492] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d: Bootstrap complete.
I20240502 14:15:18.981742 20855 ts_tablet_manager.cc:1367] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.097s user 0.002s sys 0.004s
I20240502 14:15:18.982498 20855 raft_consensus.cc:348] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:18.986277 20855 raft_consensus.cc:729] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: LEARNER
I20240502 14:15:18.986506 20855 consensus_queue.cc:260] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:18.988185 20855 ts_tablet_manager.cc:1398] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240502 14:15:18.988871 16175 tablet_copy_service.cc:342] P c86f7440570f4796babfd44c3d7fd847: Request end of tablet copy session cd67493f6f0142f896010f2b8a39887d-e7b93f8449ef4e6fb0a7e315130da024 received from {username='slave'} at 127.6.196.194:59023
I20240502 14:15:18.989022 16175 tablet_copy_service.cc:433] P c86f7440570f4796babfd44c3d7fd847: ending tablet copy session cd67493f6f0142f896010f2b8a39887d-e7b93f8449ef4e6fb0a7e315130da024 on tablet e7b93f8449ef4e6fb0a7e315130da024 with peer cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:18.992702 13040 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table [id=80ece6bedeed43b98d172a5546be930b] alter complete (version 2)
I20240502 14:15:19.005213 20894 ts_tablet_manager.cc:902] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:19.011888 16285 raft_consensus.cc:1261] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.012089 16019 raft_consensus.cc:1261] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.012715 20474 consensus_queue.cc:1035] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:19.013353 20474 consensus_queue.cc:1035] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:19.014683 13384 consensus_peers.cc:485] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c5652dfa60d84803abbac9201e5874db. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:19.043699 20824 tablet_bootstrap.cc:492] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:19.044081 20824 tablet_bootstrap.cc:492] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:15:19.044799 20837 consensus_queue.cc:1035] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.063338 20475 raft_consensus.cc:2932] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:19.066020 16023 raft_consensus.cc:2932] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:19.077613 13038 catalog_manager.cc:5484] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.087327 20824 ts_tablet_manager.cc:1367] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.399s user 0.072s sys 0.012s
I20240502 14:15:19.089565 20837 consensus_queue.cc:1035] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.092717 20824 raft_consensus.cc:348] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:19.093047 20824 raft_consensus.cc:729] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:15:19.093214 20824 consensus_queue.cc:260] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:19.093822 20824 ts_tablet_manager.cc:1398] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:19.100970 13043 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240502 14:15:19.102185 16175 tablet_copy_service.cc:342] P c86f7440570f4796babfd44c3d7fd847: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-0e393a7736b14d5483973f9cb89058f8 received from {username='slave'} at 127.6.196.193:41633
I20240502 14:15:19.102301 16175 tablet_copy_service.cc:433] P c86f7440570f4796babfd44c3d7fd847: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-0e393a7736b14d5483973f9cb89058f8 on tablet 0e393a7736b14d5483973f9cb89058f8 with peer 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:15:19.104122 16288 raft_consensus.cc:2932] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:19.108631 18885 raft_consensus.cc:482] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.108776 18885 raft_consensus.cc:504] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } }
I20240502 14:15:19.109535 18885 leader_election.cc:288] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:19.110952 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7d267297334093aa0f65b121b16b61" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:19.111188 13447 raft_consensus.cc:2379] T 4b7d267297334093aa0f65b121b16b61 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c86f7440570f4796babfd44c3d7fd847 in current term 1: Already voted for candidate d1da7f11f0604f78934354a992bb0f55 in this term.
I20240502 14:15:19.117122 20877 tablet.cc:1722] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:19.118400 20878 tablet.cc:1722] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:19.126946 16019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b7d267297334093aa0f65b121b16b61" candidate_uuid: "c86f7440570f4796babfd44c3d7fd847" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:19.128270 16090 leader_election.cc:302] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c86f7440570f4796babfd44c3d7fd847; no voters: 88b77f6edc1c4a72b548bf47c05704f3, d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:19.128638 18885 raft_consensus.cc:3037] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.129801 20880 tablet.cc:1722] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240502 14:15:19.130999 18885 raft_consensus.cc:2726] T 4b7d267297334093aa0f65b121b16b61 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:19.141682 20867 tablet_bootstrap.cc:492] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:19.141886 20788 consensus_queue.cc:1035] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.142285 20867 tablet_bootstrap.cc:492] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:15:19.142591 20867 ts_tablet_manager.cc:1367] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.166s user 0.052s sys 0.001s
I20240502 14:15:19.143381 20867 raft_consensus.cc:348] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:19.143687 20867 raft_consensus.cc:729] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:15:19.143930 20867 consensus_queue.cc:260] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:19.145881 20867 ts_tablet_manager.cc:1398] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:15:19.146440 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-0854e9f4851e400f9af18181a71ae8f7 received from {username='slave'} at 127.6.196.200:56671
I20240502 14:15:19.146566 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 8b47d4390f8343a493d6c00123785968-0854e9f4851e400f9af18181a71ae8f7 on tablet 0854e9f4851e400f9af18181a71ae8f7 with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:15:19.152531 20835 consensus_queue.cc:1035] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.158547 20788 consensus_queue.cc:1035] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.187603 13040 catalog_manager.cc:6029] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: rebalancer_dynamic_table_new_name [id=80ece6bedeed43b98d172a5546be930b] alter complete (version 3)
I20240502 14:15:19.190935 13318 consensus_queue.cc:237] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:19.194537 16154 raft_consensus.cc:1261] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:19.194940 13251 consensus_peers.cc:485] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd06e00fd93340099df5e020716033a9. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:19.195490 20393 consensus_queue.cc:1035] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:19.198707 16420 raft_consensus.cc:1261] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:19.201927 20835 consensus_queue.cc:1035] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:19.209267 20643 raft_consensus.cc:1050] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 to VOTER
I20240502 14:15:19.209478 20837 raft_consensus.cc:2932] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:19.210285 20643 consensus_queue.cc:237] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:19.213124 16154 raft_consensus.cc:2932] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:19.232913 13318 raft_consensus.cc:1203] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:19.241081 13040 catalog_manager.cc:5484] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.269767 20837 consensus_queue.cc:1035] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:19.270737 16418 raft_consensus.cc:2932] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:19.278847 13187 raft_consensus.cc:1261] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:19.279287 13447 raft_consensus.cc:1261] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:19.280778 20894 tablet_copy_client.cc:308] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:15:19.280701 20641 consensus_queue.cc:1035] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:19.281347 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet 27857198d4ec430ebee897c90141c58d from peer d1da7f11f0604f78934354a992bb0f55 ({username='slave'} at 127.6.196.197:40701)
I20240502 14:15:19.281471 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet 27857198d4ec430ebee897c90141c58d from peer d1da7f11f0604f78934354a992bb0f55 at {username='slave'} at 127.6.196.197:40701: session id = d1da7f11f0604f78934354a992bb0f55-27857198d4ec430ebee897c90141c58d
I20240502 14:15:19.281785 16154 raft_consensus.cc:1261] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:19.283150 20642 consensus_queue.cc:1035] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:19.283586 20642 consensus_queue.cc:1035] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:19.283954 13601 tablet_copy_source_session.cc:205] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:19.295296 13045 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:19.295490 13045 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39658:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240502 14:15:19.300649 20641 raft_consensus.cc:2932] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:19.301375 13045 catalog_manager.cc:5771] T 00000000000000000000000000000000 P f161cc9c91884c50874ece8743051e73: Sending DeleteTablet for 3 replicas of tablet a337c4f1550c42db8f8f46eaf600554f
I20240502 14:15:19.302538 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet a337c4f1550c42db8f8f46eaf600554f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:19 UTC) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:19.303015 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet a337c4f1550c42db8f8f46eaf600554f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:19 UTC) from {username='slave'} at 127.0.0.1:57492
I20240502 14:15:19.310382 13186 raft_consensus.cc:2932] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:19.312326 16419 raft_consensus.cc:1203] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:15:19.319727 16152 raft_consensus.cc:2932] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:19.326247 20927 tablet_replica.cc:330] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:19.326462 20927 raft_consensus.cc:2229] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:19.326685 20927 raft_consensus.cc:2258] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:19.327980 13038 catalog_manager.cc:5484] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:19.329641 20927 ts_tablet_manager.cc:1875] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:19.341653 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet a337c4f1550c42db8f8f46eaf600554f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-02 14:15:19 UTC) from {username='slave'} at 127.0.0.1:58526
I20240502 14:15:19.345788 20927 ts_tablet_manager.cc:1888] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:19.346040 20927 log.cc:1196] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/a337c4f1550c42db8f8f46eaf600554f
I20240502 14:15:19.347790 20867 ts_tablet_manager.cc:902] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:19.348199 20927 ts_tablet_manager.cc:1909] T a337c4f1550c42db8f8f46eaf600554f P 8b47d4390f8343a493d6c00123785968: Deleting consensus metadata
I20240502 14:15:19.349246 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet a337c4f1550c42db8f8f46eaf600554f (table rebalancer_dynamic_table_new_name [id=80ece6bedeed43b98d172a5546be930b]) successfully deleted
I20240502 14:15:19.364987 20894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27857198d4ec430ebee897c90141c58d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.384271 13039 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_extra_table_54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:19.387701 20928 tablet_replica.cc:330] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:15:19.398914 13166 tablet_service.cc:1467] Processing CreateTablet for tablet 5661c0f4fe074dd28e81d195d6607960 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=180fa1d6120045839986b6084e586870]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:19.399322 13166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5661c0f4fe074dd28e81d195d6607960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.399480 16401 tablet_service.cc:1467] Processing CreateTablet for tablet 5661c0f4fe074dd28e81d195d6607960 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=180fa1d6120045839986b6084e586870]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:19.399916 16401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5661c0f4fe074dd28e81d195d6607960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.398536 20930 tablet_replica.cc:330] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:15:19.405478 20930 raft_consensus.cc:2229] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Raft consensus shutting down.
I20240502 14:15:19.406533 16002 tablet_service.cc:1467] Processing CreateTablet for tablet 5661c0f4fe074dd28e81d195d6607960 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=180fa1d6120045839986b6084e586870]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:19.406949 16002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5661c0f4fe074dd28e81d195d6607960. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.407289 16399 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8443bed0da49d0b24511a8468c2373 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=180fa1d6120045839986b6084e586870]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:19.407359 16265 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8443bed0da49d0b24511a8468c2373 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=180fa1d6120045839986b6084e586870]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:19.407706 16399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8443bed0da49d0b24511a8468c2373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.407745 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8443bed0da49d0b24511a8468c2373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.408572 13560 tablet_service.cc:1467] Processing CreateTablet for tablet 57b8b76eff294d42a45e3c9240ff23c8 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=180fa1d6120045839986b6084e586870]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:19.408939 13560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b8b76eff294d42a45e3c9240ff23c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.409504 13168 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8443bed0da49d0b24511a8468c2373 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=180fa1d6120045839986b6084e586870]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:19.409579 16400 tablet_service.cc:1467] Processing CreateTablet for tablet 57b8b76eff294d42a45e3c9240ff23c8 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=180fa1d6120045839986b6084e586870]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:19.409801 13168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8443bed0da49d0b24511a8468c2373. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.409934 16400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b8b76eff294d42a45e3c9240ff23c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.415778 16266 tablet_service.cc:1467] Processing CreateTablet for tablet 57b8b76eff294d42a45e3c9240ff23c8 (DEFAULT_TABLE table=rebalancer_extra_table_54 [id=180fa1d6120045839986b6084e586870]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:19.416168 16266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b8b76eff294d42a45e3c9240ff23c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.430225 13447 raft_consensus.cc:2932] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:19.432262 20928 raft_consensus.cc:2229] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:19.432469 20928 raft_consensus.cc:2258] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:19.434376 13580 consensus_queue.cc:237] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:19.434659 20930 raft_consensus.cc:2258] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:19.435562 20930 ts_tablet_manager.cc:1875] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:19.437279 13186 raft_consensus.cc:1261] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:19.438560 20928 ts_tablet_manager.cc:1875] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240502 14:15:19.438788 20641 consensus_queue.cc:1035] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:19.443243 13039 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39658:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240502 14:15:19.445252 20923 raft_consensus.cc:2932] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:19.449678 20942 tablet_bootstrap.cc:492] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:19.450511 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 861f9171cb654ec889ddf824f2bb6e92 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:19.451200 13186 raft_consensus.cc:2932] T 861f9171cb654ec889ddf824f2bb6e92 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:19.453691 13040 catalog_manager.cc:5484] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.454484 20894 tablet_copy_client.cc:783] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:19.454516 20942 tablet_bootstrap.cc:654] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.454758 20894 tablet_copy_client.cc:647] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:19.457453 16133 tablet_service.cc:1514] Processing DeleteTablet for tablet 861f9171cb654ec889ddf824f2bb6e92 with delete_type TABLET_DATA_TOMBSTONED (TS c86f7440570f4796babfd44c3d7fd847 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59110
I20240502 14:15:19.463446 16265 tablet_service.cc:1467] Processing CreateTablet for tablet e62e66aad6d24367a4245e84359a490b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=83c5268a455d403b843aee90a2f6b033]), partition=
I20240502 14:15:19.463426 13427 tablet_service.cc:1467] Processing CreateTablet for tablet e62e66aad6d24367a4245e84359a490b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=83c5268a455d403b843aee90a2f6b033]), partition=
I20240502 14:15:19.463776 13427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62e66aad6d24367a4245e84359a490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.463909 16265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62e66aad6d24367a4245e84359a490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.470615 20943 tablet_replica.cc:330] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847: stopping tablet replica
I20240502 14:15:19.470877 20943 raft_consensus.cc:2229] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:19.471252 20943 raft_consensus.cc:2258] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:19.472837 13445 raft_consensus.cc:1261] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:19.473685 20929 consensus_queue.cc:1035] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:19.474331 20939 tablet_bootstrap.cc:492] T 5661c0f4fe074dd28e81d195d6607960 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:19.475021 20894 tablet_copy_client.cc:523] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:19.475579 13559 tablet_service.cc:1467] Processing CreateTablet for tablet e62e66aad6d24367a4245e84359a490b (DEFAULT_TABLE table=rebalancer_dynamic_table [id=83c5268a455d403b843aee90a2f6b033]), partition=
I20240502 14:15:19.477562 20943 ts_tablet_manager.cc:1875] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:19.478453 20894 tablet_bootstrap.cc:492] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:19.478906 20939 tablet_bootstrap.cc:654] T 5661c0f4fe074dd28e81d195d6607960 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.479475 13559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e62e66aad6d24367a4245e84359a490b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.480329 20937 tablet_bootstrap.cc:492] T 5661c0f4fe074dd28e81d195d6607960 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:19.485666 20940 tablet_bootstrap.cc:492] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55: Bootstrap starting.
I20240502 14:15:19.488345 13186 raft_consensus.cc:1203] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240502 14:15:19.488584 20940 tablet_bootstrap.cc:654] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.492326 20943 ts_tablet_manager.cc:1888] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:19.492489 20943 log.cc:1196] T 861f9171cb654ec889ddf824f2bb6e92 P c86f7440570f4796babfd44c3d7fd847: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/wals/861f9171cb654ec889ddf824f2bb6e92
I20240502 14:15:19.494082 13030 catalog_manager.cc:4830] TS c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): tablet 861f9171cb654ec889ddf824f2bb6e92 (table rebalancer_extra_table_22 [id=24518149b1dc474687818a2a77d62dda]) successfully deleted
I20240502 14:15:19.494159 20944 tablet_bootstrap.cc:492] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:19.494230 20930 ts_tablet_manager.cc:1888] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:19.494572 20930 log.cc:1196] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/a337c4f1550c42db8f8f46eaf600554f
I20240502 14:15:19.495256 20930 ts_tablet_manager.cc:1909] T a337c4f1550c42db8f8f46eaf600554f P 7eb4ecb2379b44c0954f2658c4a04350: Deleting consensus metadata
I20240502 14:15:19.496268 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet a337c4f1550c42db8f8f46eaf600554f (table rebalancer_dynamic_table_new_name [id=80ece6bedeed43b98d172a5546be930b]) successfully deleted
I20240502 14:15:19.498309 20944 tablet_bootstrap.cc:654] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.507450 20938 tablet_bootstrap.cc:492] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:19.510553 20944 tablet_bootstrap.cc:492] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3: No bootstrap required, opened a new log
I20240502 14:15:19.511157 20937 tablet_bootstrap.cc:654] T 5661c0f4fe074dd28e81d195d6607960 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.511603 20928 ts_tablet_manager.cc:1888] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240502 14:15:19.511930 20867 tablet_copy_client.cc:308] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.197:43695
I20240502 14:15:19.512461 20938 tablet_bootstrap.cc:654] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.512476 20928 log.cc:1196] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/a337c4f1550c42db8f8f46eaf600554f
I20240502 14:15:19.512914 20928 ts_tablet_manager.cc:1909] T a337c4f1550c42db8f8f46eaf600554f P cd67493f6f0142f896010f2b8a39887d: Deleting consensus metadata
I20240502 14:15:19.513031 16043 tablet_copy_service.cc:140] P d1da7f11f0604f78934354a992bb0f55: Received BeginTabletCopySession request for tablet fae5b25ec9694cea87381d27f4d46384 from peer 8b47d4390f8343a493d6c00123785968 ({username='slave'} at 127.6.196.200:54603)
I20240502 14:15:19.513221 16043 tablet_copy_service.cc:161] P d1da7f11f0604f78934354a992bb0f55: Beginning new tablet copy session on tablet fae5b25ec9694cea87381d27f4d46384 from peer 8b47d4390f8343a493d6c00123785968 at {username='slave'} at 127.6.196.200:54603: session id = 8b47d4390f8343a493d6c00123785968-fae5b25ec9694cea87381d27f4d46384
I20240502 14:15:19.515838 16043 tablet_copy_source_session.cc:205] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:19.515903 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet a337c4f1550c42db8f8f46eaf600554f (table rebalancer_dynamic_table_new_name [id=80ece6bedeed43b98d172a5546be930b]) successfully deleted
I20240502 14:15:19.518252 20867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fae5b25ec9694cea87381d27f4d46384. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.521554 20944 ts_tablet_manager.cc:1367] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20240502 14:15:19.523536 20940 tablet_bootstrap.cc:492] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55: No bootstrap required, opened a new log
I20240502 14:15:19.523718 20940 ts_tablet_manager.cc:1367] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.000s
I20240502 14:15:19.525022 20944 raft_consensus.cc:348] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.525050 20940 raft_consensus.cc:348] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:19.525206 20944 raft_consensus.cc:374] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.525242 20940 raft_consensus.cc:374] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.525310 20944 raft_consensus.cc:729] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.525327 20940 raft_consensus.cc:729] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.525513 20944 consensus_queue.cc:260] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.525565 20940 consensus_queue.cc:260] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:19.526189 20940 ts_tablet_manager.cc:1398] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:19.526985 13447 raft_consensus.cc:2932] T 861f9171cb654ec889ddf824f2bb6e92 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:19.536372 20942 tablet_bootstrap.cc:492] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:19.536562 20942 ts_tablet_manager.cc:1367] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.087s user 0.007s sys 0.007s
I20240502 14:15:19.537457 20942 raft_consensus.cc:348] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:19.537711 20942 raft_consensus.cc:374] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.537834 20942 raft_consensus.cc:729] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.539202 20942 consensus_queue.cc:260] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:19.540385 20944 ts_tablet_manager.cc:1398] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20240502 14:15:19.548353 20867 tablet_copy_client.cc:783] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:19.548631 20867 tablet_copy_client.cc:647] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:19.551959 20867 tablet_copy_client.cc:523] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:19.556000 20938 tablet_bootstrap.cc:492] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:19.556178 20938 ts_tablet_manager.cc:1367] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.000s
I20240502 14:15:19.556246 20942 ts_tablet_manager.cc:1398] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240502 14:15:19.557885 20867 tablet_bootstrap.cc:492] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:19.558580 20937 tablet_bootstrap.cc:492] T 5661c0f4fe074dd28e81d195d6607960 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:19.558806 20937 ts_tablet_manager.cc:1367] T 5661c0f4fe074dd28e81d195d6607960 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.004s
I20240502 14:15:19.559736 20939 tablet_bootstrap.cc:492] T 5661c0f4fe074dd28e81d195d6607960 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:19.560832 20942 tablet_bootstrap.cc:492] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:19.561764 20939 ts_tablet_manager.cc:1367] T 5661c0f4fe074dd28e81d195d6607960 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.088s user 0.003s sys 0.003s
I20240502 14:15:19.562500 20937 raft_consensus.cc:348] T 5661c0f4fe074dd28e81d195d6607960 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:19.562700 20942 tablet_bootstrap.cc:654] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.563283 20938 raft_consensus.cc:348] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.564224 20939 raft_consensus.cc:348] T 5661c0f4fe074dd28e81d195d6607960 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:19.567735 20937 raft_consensus.cc:374] T 5661c0f4fe074dd28e81d195d6607960 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.567888 20938 raft_consensus.cc:374] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.568050 20937 raft_consensus.cc:729] T 5661c0f4fe074dd28e81d195d6607960 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.568150 20938 raft_consensus.cc:729] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.568315 20939 raft_consensus.cc:374] T 5661c0f4fe074dd28e81d195d6607960 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.568542 20938 consensus_queue.cc:260] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.568895 20937 consensus_queue.cc:260] T 5661c0f4fe074dd28e81d195d6607960 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:19.569255 20939 raft_consensus.cc:729] T 5661c0f4fe074dd28e81d195d6607960 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.569681 20939 consensus_queue.cc:260] T 5661c0f4fe074dd28e81d195d6607960 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:19.570151 20954 ts_tablet_manager.cc:902] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d: Initiating tablet copy from peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:19.578481 20937 ts_tablet_manager.cc:1398] T 5661c0f4fe074dd28e81d195d6607960 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.017s user 0.000s sys 0.001s
I20240502 14:15:19.578904 20938 ts_tablet_manager.cc:1398] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20240502 14:15:19.584501 20643 raft_consensus.cc:482] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.584618 20643 raft_consensus.cc:504] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.585420 20643 leader_election.cc:288] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:19.586655 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b8b76eff294d42a45e3c9240ff23c8" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:19.587082 16285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b8b76eff294d42a45e3c9240ff23c8" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
W20240502 14:15:19.588876 13516 leader_election.cc:341] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Illegal state: must be running to vote when last-logged opid is not known
W20240502 14:15:19.590152 13515 leader_election.cc:341] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Illegal state: must be running to vote when last-logged opid is not known
I20240502 14:15:19.590265 13515 leader_election.cc:302] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:19.590988 20929 raft_consensus.cc:2726] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:19.592661 20942 tablet_bootstrap.cc:492] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:19.592849 20942 ts_tablet_manager.cc:1367] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.002s
I20240502 14:15:19.593958 20942 raft_consensus.cc:348] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.594236 20942 raft_consensus.cc:374] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.594347 20942 raft_consensus.cc:729] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.594583 20942 consensus_queue.cc:260] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.596362 20942 ts_tablet_manager.cc:1398] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:15:19.596628 20942 tablet_bootstrap.cc:492] T e62e66aad6d24367a4245e84359a490b P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:19.598138 20938 tablet_bootstrap.cc:492] T e62e66aad6d24367a4245e84359a490b P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:19.598405 20937 tablet_bootstrap.cc:492] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:19.598620 20939 ts_tablet_manager.cc:1398] T 5661c0f4fe074dd28e81d195d6607960 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.035s user 0.000s sys 0.002s
I20240502 14:15:19.598672 20942 tablet_bootstrap.cc:654] T e62e66aad6d24367a4245e84359a490b P 5297e816f782430d9a01220f0471a882: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.604182 20939 tablet_bootstrap.cc:492] T 1d8443bed0da49d0b24511a8468c2373 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:19.606217 20937 tablet_bootstrap.cc:654] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.608829 20938 tablet_bootstrap.cc:654] T e62e66aad6d24367a4245e84359a490b P 7eb4ecb2379b44c0954f2658c4a04350: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.609397 20939 tablet_bootstrap.cc:654] T 1d8443bed0da49d0b24511a8468c2373 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.614409 20942 tablet_bootstrap.cc:492] T e62e66aad6d24367a4245e84359a490b P 5297e816f782430d9a01220f0471a882: No bootstrap required, opened a new log
I20240502 14:15:19.614581 20942 ts_tablet_manager.cc:1367] T e62e66aad6d24367a4245e84359a490b P 5297e816f782430d9a01220f0471a882: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.007s
I20240502 14:15:19.616437 20937 tablet_bootstrap.cc:492] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625: No bootstrap required, opened a new log
I20240502 14:15:19.616641 20937 ts_tablet_manager.cc:1367] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240502 14:15:19.617223 20942 raft_consensus.cc:348] T e62e66aad6d24367a4245e84359a490b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.617714 20942 raft_consensus.cc:374] T e62e66aad6d24367a4245e84359a490b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.617882 20942 raft_consensus.cc:729] T e62e66aad6d24367a4245e84359a490b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5297e816f782430d9a01220f0471a882, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.618095 20942 consensus_queue.cc:260] T e62e66aad6d24367a4245e84359a490b P 5297e816f782430d9a01220f0471a882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.618719 20937 raft_consensus.cc:348] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:19.618896 20937 raft_consensus.cc:374] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.618940 20937 raft_consensus.cc:729] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.621246 20938 tablet_bootstrap.cc:492] T e62e66aad6d24367a4245e84359a490b P 7eb4ecb2379b44c0954f2658c4a04350: No bootstrap required, opened a new log
I20240502 14:15:19.622480 20939 tablet_bootstrap.cc:492] T 1d8443bed0da49d0b24511a8468c2373 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:19.622651 20938 ts_tablet_manager.cc:1367] T e62e66aad6d24367a4245e84359a490b P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20240502 14:15:19.622957 20942 ts_tablet_manager.cc:1398] T e62e66aad6d24367a4245e84359a490b P 5297e816f782430d9a01220f0471a882: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240502 14:15:19.623251 20939 ts_tablet_manager.cc:1367] T 1d8443bed0da49d0b24511a8468c2373 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20240502 14:15:19.624320 18885 raft_consensus.cc:1050] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847: attempting to promote NON_VOTER cd67493f6f0142f896010f2b8a39887d to VOTER
I20240502 14:15:19.624087 20867 tablet_bootstrap.cc:492] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:19.625180 20867 tablet_bootstrap.cc:492] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968: Bootstrap complete.
I20240502 14:15:19.625030 18885 consensus_queue.cc:237] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:19.625423 20867 ts_tablet_manager.cc:1367] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.068s user 0.013s sys 0.041s
I20240502 14:15:19.626458 20867 raft_consensus.cc:348] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:19.626674 20867 raft_consensus.cc:729] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: LEARNER
I20240502 14:15:19.626849 20867 consensus_queue.cc:260] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:19.628973 20867 ts_tablet_manager.cc:1398] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:19.629940 20939 raft_consensus.cc:348] T 1d8443bed0da49d0b24511a8468c2373 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:19.630158 20939 raft_consensus.cc:374] T 1d8443bed0da49d0b24511a8468c2373 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.630234 20939 raft_consensus.cc:729] T 1d8443bed0da49d0b24511a8468c2373 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.630877 16043 tablet_copy_service.cc:342] P d1da7f11f0604f78934354a992bb0f55: Request end of tablet copy session 8b47d4390f8343a493d6c00123785968-fae5b25ec9694cea87381d27f4d46384 received from {username='slave'} at 127.6.196.200:54603
I20240502 14:15:19.631018 16043 tablet_copy_service.cc:433] P d1da7f11f0604f78934354a992bb0f55: ending tablet copy session 8b47d4390f8343a493d6c00123785968-fae5b25ec9694cea87381d27f4d46384 on tablet fae5b25ec9694cea87381d27f4d46384 with peer 8b47d4390f8343a493d6c00123785968
I20240502 14:15:19.623477 20937 consensus_queue.cc:260] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:19.636950 20937 ts_tablet_manager.cc:1398] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240502 14:15:19.637032 20938 raft_consensus.cc:348] T e62e66aad6d24367a4245e84359a490b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.639403 20954 tablet_copy_client.cc:308] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.195:46303
I20240502 14:15:19.639701 20938 raft_consensus.cc:374] T e62e66aad6d24367a4245e84359a490b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.639977 20938 raft_consensus.cc:729] T e62e66aad6d24367a4245e84359a490b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.641072 13470 tablet_copy_service.cc:140] P 88b77f6edc1c4a72b548bf47c05704f3: Received BeginTabletCopySession request for tablet c5652dfa60d84803abbac9201e5874db from peer cd67493f6f0142f896010f2b8a39887d ({username='slave'} at 127.6.196.194:48133)
I20240502 14:15:19.641197 13470 tablet_copy_service.cc:161] P 88b77f6edc1c4a72b548bf47c05704f3: Beginning new tablet copy session on tablet c5652dfa60d84803abbac9201e5874db from peer cd67493f6f0142f896010f2b8a39887d at {username='slave'} at 127.6.196.194:48133: session id = cd67493f6f0142f896010f2b8a39887d-c5652dfa60d84803abbac9201e5874db
I20240502 14:15:19.643697 20938 consensus_queue.cc:260] T e62e66aad6d24367a4245e84359a490b P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.643900 13470 tablet_copy_source_session.cc:205] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:19.644577 20610 raft_consensus.cc:482] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.644726 20610 raft_consensus.cc:504] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:19.645439 20610 leader_election.cc:288] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:19.645947 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8443bed0da49d0b24511a8468c2373" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:19.646227 13185 raft_consensus.cc:2454] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:19.646961 16219 leader_election.cc:302] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 5297e816f782430d9a01220f0471a882; no voters:
I20240502 14:15:19.651144 20938 ts_tablet_manager.cc:1398] T e62e66aad6d24367a4245e84359a490b P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.028s user 0.001s sys 0.000s
I20240502 14:15:19.644019 20939 consensus_queue.cc:260] T 1d8443bed0da49d0b24511a8468c2373 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:19.647456 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8443bed0da49d0b24511a8468c2373" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:19.652056 16419 raft_consensus.cc:2454] T 1d8443bed0da49d0b24511a8468c2373 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:19.653674 20788 raft_consensus.cc:482] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.653810 20788 raft_consensus.cc:504] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:19.654433 20788 leader_election.cc:288] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:19.654886 20610 raft_consensus.cc:2781] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.654986 20610 raft_consensus.cc:482] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.655081 20610 raft_consensus.cc:3037] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.655275 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5661c0f4fe074dd28e81d195d6607960" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:19.655417 13185 raft_consensus.cc:2454] T 5661c0f4fe074dd28e81d195d6607960 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:19.655745 15955 leader_election.cc:302] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:19.656040 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5661c0f4fe074dd28e81d195d6607960" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:19.656205 16419 raft_consensus.cc:2454] T 5661c0f4fe074dd28e81d195d6607960 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 0.
I20240502 14:15:19.658010 20788 raft_consensus.cc:2781] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.658128 20788 raft_consensus.cc:482] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.658215 20788 raft_consensus.cc:3037] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.652034 20939 ts_tablet_manager.cc:1398] T 1d8443bed0da49d0b24511a8468c2373 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.023s user 0.000s sys 0.001s
I20240502 14:15:19.658752 20939 tablet_bootstrap.cc:492] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968: Bootstrap starting.
I20240502 14:15:19.661017 20788 raft_consensus.cc:504] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:19.660845 20610 raft_consensus.cc:504] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:19.663048 20610 leader_election.cc:288] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:19.663493 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5661c0f4fe074dd28e81d195d6607960" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:19.663677 13187 raft_consensus.cc:3037] T 5661c0f4fe074dd28e81d195d6607960 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.664471 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8443bed0da49d0b24511a8468c2373" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:19.664615 16419 raft_consensus.cc:3037] T 1d8443bed0da49d0b24511a8468c2373 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.665177 20939 tablet_bootstrap.cc:654] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:19.666020 13187 raft_consensus.cc:2454] T 5661c0f4fe074dd28e81d195d6607960 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:19.666468 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8443bed0da49d0b24511a8468c2373" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:19.666597 13185 raft_consensus.cc:3037] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.667068 16419 raft_consensus.cc:2454] T 1d8443bed0da49d0b24511a8468c2373 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:19.668678 16220 leader_election.cc:302] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:19.668792 13185 raft_consensus.cc:2454] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:19.669242 20610 raft_consensus.cc:2781] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.669489 15955 leader_election.cc:302] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:19.669514 20610 raft_consensus.cc:686] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:19.669772 20609 raft_consensus.cc:2781] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.669795 20610 consensus_queue.cc:237] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:19.669948 20609 raft_consensus.cc:686] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:19.670184 20609 consensus_queue.cc:237] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:19.670385 16419 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5661c0f4fe074dd28e81d195d6607960" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:19.670589 16419 raft_consensus.cc:3037] T 5661c0f4fe074dd28e81d195d6607960 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.671419 20788 leader_election.cc:288] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 1 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:19.672287 20954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5652dfa60d84803abbac9201e5874db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.673450 16419 raft_consensus.cc:2454] T 5661c0f4fe074dd28e81d195d6607960 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:19.675833 16285 raft_consensus.cc:1261] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:19.676333 20603 consensus_queue.cc:1035] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:19.683916 13319 raft_consensus.cc:1261] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:19.684595 20954 tablet_copy_client.cc:783] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:19.684566 20603 consensus_queue.cc:1035] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:19.684767 20954 tablet_copy_client.cc:647] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:19.687208 16018 raft_consensus.cc:1261] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:19.687798 20603 consensus_queue.cc:1035] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:19.691213 20954 tablet_copy_client.cc:523] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:19.696992 20954 tablet_bootstrap.cc:492] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
W20240502 14:15:19.703969 20966 log.cc:924] Time spent T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847: Append to log took a long time: real 0.050s user 0.000s sys 0.000s
I20240502 14:15:19.706812 20603 raft_consensus.cc:2932] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:19.707367 13040 catalog_manager.cc:5484] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 0 to 1, leader changed from <none> to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:19.707922 20939 tablet_bootstrap.cc:492] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968: No bootstrap required, opened a new log
I20240502 14:15:19.708114 20939 ts_tablet_manager.cc:1367] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968: Time spent bootstrapping tablet: real 0.050s user 0.006s sys 0.007s
I20240502 14:15:19.711827 20939 raft_consensus.cc:348] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.712455 20939 raft_consensus.cc:374] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:19.712579 20939 raft_consensus.cc:729] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Initialized, Role: FOLLOWER
I20240502 14:15:19.712347 16018 raft_consensus.cc:2932] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:19.712843 20939 consensus_queue.cc:260] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.713567 20939 ts_tablet_manager.cc:1398] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:15:19.714447 20969 ts_tablet_manager.cc:902] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:19.717540 13039 catalog_manager.cc:5484] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.720094 16285 raft_consensus.cc:2932] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:19.720240 13319 raft_consensus.cc:2932] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:19.727178 13040 catalog_manager.cc:5484] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:19.736919 20954 tablet_bootstrap.cc:492] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:19.737345 20954 tablet_bootstrap.cc:492] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d: Bootstrap complete.
I20240502 14:15:19.737653 20954 ts_tablet_manager.cc:1367] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.041s user 0.010s sys 0.000s
I20240502 14:15:19.738795 20954 raft_consensus.cc:348] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:19.739054 20954 raft_consensus.cc:729] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: LEARNER
I20240502 14:15:19.739356 20954 consensus_queue.cc:260] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:19.739878 16151 consensus_queue.cc:237] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:19.740358 20954 ts_tablet_manager.cc:1398] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:19.741153 13470 tablet_copy_service.cc:342] P 88b77f6edc1c4a72b548bf47c05704f3: Request end of tablet copy session cd67493f6f0142f896010f2b8a39887d-c5652dfa60d84803abbac9201e5874db received from {username='slave'} at 127.6.196.194:48133
I20240502 14:15:19.741286 13470 tablet_copy_service.cc:433] P 88b77f6edc1c4a72b548bf47c05704f3: ending tablet copy session cd67493f6f0142f896010f2b8a39887d-c5652dfa60d84803abbac9201e5874db on tablet c5652dfa60d84803abbac9201e5874db with peer cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:19.746590 16285 raft_consensus.cc:1261] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:19.747123 20603 consensus_queue.cc:1035] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:19.748694 13319 raft_consensus.cc:1261] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:19.749311 20603 consensus_queue.cc:1035] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:19.751091 18885 raft_consensus.cc:2932] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:19.756646 13319 raft_consensus.cc:2932] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:19.759390 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e7b93f8449ef4e6fb0a7e315130da024 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:19.762537 16285 raft_consensus.cc:2932] T e7b93f8449ef4e6fb0a7e315130da024 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:19.764412 13039 catalog_manager.cc:5484] T e7b93f8449ef4e6fb0a7e315130da024 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:19.768514 16002 tablet_service.cc:1514] Processing DeleteTablet for tablet e7b93f8449ef4e6fb0a7e315130da024 with delete_type TABLET_DATA_TOMBSTONED (TS d1da7f11f0604f78934354a992bb0f55 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:19.771597 20988 tablet_replica.cc:330] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:19.771934 20988 raft_consensus.cc:2229] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:19.772507 20988 raft_consensus.cc:2258] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:19.772809 20969 tablet_copy_client.cc:308] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:15:19.773550 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet cd06e00fd93340099df5e020716033a9 from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:32917)
I20240502 14:15:19.773679 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet cd06e00fd93340099df5e020716033a9 from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:32917: session id = 20a967bad5504b1e9c6fb720ecdd1625-cd06e00fd93340099df5e020716033a9
I20240502 14:15:19.776623 13339 tablet_copy_source_session.cc:205] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:19.778349 20969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd06e00fd93340099df5e020716033a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:19.782968 20988 ts_tablet_manager.cc:1875] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:19.784726 20894 tablet_bootstrap.cc:492] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:19.785184 20894 tablet_bootstrap.cc:492] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55: Bootstrap complete.
I20240502 14:15:19.785461 20894 ts_tablet_manager.cc:1367] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55: Time spent bootstrapping tablet: real 0.307s user 0.053s sys 0.005s
I20240502 14:15:19.786221 20894 raft_consensus.cc:348] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:19.786479 20894 raft_consensus.cc:729] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Initialized, Role: LEARNER
I20240502 14:15:19.786665 20894 consensus_queue.cc:260] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: NON_VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: true } }
I20240502 14:15:19.787180 20894 ts_tablet_manager.cc:1398] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:19.790055 20969 tablet_copy_client.cc:783] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:19.790369 20969 tablet_copy_client.cc:647] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:19.791522 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session d1da7f11f0604f78934354a992bb0f55-27857198d4ec430ebee897c90141c58d received from {username='slave'} at 127.6.196.197:40701
I20240502 14:15:19.791674 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session d1da7f11f0604f78934354a992bb0f55-27857198d4ec430ebee897c90141c58d on tablet 27857198d4ec430ebee897c90141c58d with peer d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:19.791898 20969 tablet_copy_client.cc:523] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:19.797176 20969 tablet_bootstrap.cc:492] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:19.806151 20988 ts_tablet_manager.cc:1888] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:19.806440 20988 log.cc:1196] T e7b93f8449ef4e6fb0a7e315130da024 P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/e7b93f8449ef4e6fb0a7e315130da024
I20240502 14:15:19.807533 20969 tablet_bootstrap.cc:492] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:19.807878 20969 tablet_bootstrap.cc:492] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:15:19.808071 20969 ts_tablet_manager.cc:1367] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240502 14:15:19.808662 20969 raft_consensus.cc:348] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:19.808867 20969 raft_consensus.cc:729] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:15:19.809019 20969 consensus_queue.cc:260] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:19.809298 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet e7b93f8449ef4e6fb0a7e315130da024 (table rebalancer_extra_table_22 [id=24518149b1dc474687818a2a77d62dda]) successfully deleted
I20240502 14:15:19.811172 20969 ts_tablet_manager.cc:1398] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:19.811733 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-cd06e00fd93340099df5e020716033a9 received from {username='slave'} at 127.6.196.193:32917
I20240502 14:15:19.811832 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-cd06e00fd93340099df5e020716033a9 on tablet cd06e00fd93340099df5e020716033a9 with peer 20a967bad5504b1e9c6fb720ecdd1625
W20240502 14:15:19.822207 16353 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.195:46303, user_credentials={real_user=slave}} blocked reactor thread for 62896.6us
I20240502 14:15:19.827131 20474 raft_consensus.cc:482] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.827256 20474 raft_consensus.cc:504] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.827903 20474 leader_election.cc:288] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:19.828509 16285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62e66aad6d24367a4245e84359a490b" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882" is_pre_election: true
I20240502 14:15:19.828732 16285 raft_consensus.cc:2454] T e62e66aad6d24367a4245e84359a490b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:15:19.829357 13384 leader_election.cc:302] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:19.830060 20474 raft_consensus.cc:2781] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.830147 20474 raft_consensus.cc:482] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.830090 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62e66aad6d24367a4245e84359a490b" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:19.830214 20474 raft_consensus.cc:3037] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.830272 13580 raft_consensus.cc:2454] T e62e66aad6d24367a4245e84359a490b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 0.
I20240502 14:15:19.834765 20474 raft_consensus.cc:504] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.835733 16285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62e66aad6d24367a4245e84359a490b" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5297e816f782430d9a01220f0471a882"
I20240502 14:15:19.835796 20474 leader_election.cc:288] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:19.835924 16285 raft_consensus.cc:3037] T e62e66aad6d24367a4245e84359a490b P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.838572 16285 raft_consensus.cc:2454] T e62e66aad6d24367a4245e84359a490b P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:19.839160 13384 leader_election.cc:302] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:19.839165 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e62e66aad6d24367a4245e84359a490b" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:19.839330 13580 raft_consensus.cc:3037] T e62e66aad6d24367a4245e84359a490b P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.839504 20474 raft_consensus.cc:2781] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.841486 20474 raft_consensus.cc:686] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:19.841928 13580 raft_consensus.cc:2454] T e62e66aad6d24367a4245e84359a490b P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:19.841888 20474 consensus_queue.cc:237] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.846586 13039 catalog_manager.cc:5484] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.898460 20393 raft_consensus.cc:1050] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:15:19.899345 20393 consensus_queue.cc:237] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:19.913540 16420 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:19.913988 16285 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:19.914299 13580 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:19.914602 20837 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:19.915268 20837 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:19.927206 16420 raft_consensus.cc:1203] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:19.932299 19959 raft_consensus.cc:482] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:19.932504 19959 raft_consensus.cc:504] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.932735 20393 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:19.933255 19959 leader_election.cc:288] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:19.933876 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b8b76eff294d42a45e3c9240ff23c8" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968" is_pre_election: true
I20240502 14:15:19.933980 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b8b76eff294d42a45e3c9240ff23c8" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:19.934134 16420 raft_consensus.cc:2454] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:19.934233 13580 raft_consensus.cc:2454] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 0.
I20240502 14:15:19.934651 16219 leader_election.cc:302] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:15:19.935256 20610 raft_consensus.cc:2781] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:19.935374 20610 raft_consensus.cc:482] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:19.935444 20610 raft_consensus.cc:3037] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.946244 20610 raft_consensus.cc:504] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.956779 16420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b8b76eff294d42a45e3c9240ff23c8" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8b47d4390f8343a493d6c00123785968"
I20240502 14:15:19.957070 16420 raft_consensus.cc:3037] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.960847 13318 raft_consensus.cc:1203] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:19.962435 16420 raft_consensus.cc:2454] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:19.963063 16220 leader_election.cc:302] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968; no voters:
I20240502 14:15:19.963474 19959 raft_consensus.cc:2781] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:19.963999 19959 raft_consensus.cc:686] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Becoming Leader. State: Replica: 5297e816f782430d9a01220f0471a882, State: Running, Role: LEADER
I20240502 14:15:19.964286 19959 consensus_queue.cc:237] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:19.965687 20610 leader_election.cc:288] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [CANDIDATE]: Term 1 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:19.966615 13580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "57b8b76eff294d42a45e3c9240ff23c8" candidate_uuid: "5297e816f782430d9a01220f0471a882" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:19.966794 13580 raft_consensus.cc:3037] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:19.971211 13580 raft_consensus.cc:2454] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5297e816f782430d9a01220f0471a882 in term 1.
I20240502 14:15:19.978196 13045 catalog_manager.cc:5484] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5297e816f782430d9a01220f0471a882 (127.6.196.199). New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:19.979876 20837 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:19.982683 16418 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:19.988912 13039 catalog_manager.cc:5484] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:19.992878 13580 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:19.997992 16288 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:20.018395 13319 consensus_queue.cc:237] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:20.020496 16420 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:20.021533 21012 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:20.025393 16019 raft_consensus.cc:1203] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240502 14:15:20.028702 16288 raft_consensus.cc:1261] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:20.029170 21012 consensus_queue.cc:1035] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:20.032182 21012 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:20.033601 16420 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:20.035841 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0854e9f4851e400f9af18181a71ae8f7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:20.037801 16288 raft_consensus.cc:2932] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:20.046216 13045 catalog_manager.cc:5484] T 0854e9f4851e400f9af18181a71ae8f7 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 8 to 9, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:20.051617 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet 0854e9f4851e400f9af18181a71ae8f7 with delete_type TABLET_DATA_TOMBSTONED (TS 7eb4ecb2379b44c0954f2658c4a04350 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:58526
I20240502 14:15:20.056166 20788 consensus_queue.cc:1035] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:20.066233 20920 raft_consensus.cc:1050] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847: attempting to promote NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 to VOTER
I20240502 14:15:20.066995 20920 consensus_queue.cc:237] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:20.075479 13186 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 LEARNER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:15:20.075848 13318 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:15:20.076421 21026 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:20.077394 13581 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240502 14:15:20.077944 18885 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:20.078200 20920 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:20.088963 13187 raft_consensus.cc:1203] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:20.141872 21023 tablet_replica.cc:330] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:15:20.142226 21023 raft_consensus.cc:2229] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:20.142608 21023 raft_consensus.cc:2258] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:20.143956 21023 ts_tablet_manager.cc:1875] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:20.154119 19959 consensus_queue.cc:1035] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:20.154892 18885 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.160058 13581 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.169473 13186 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.181020 13039 catalog_manager.cc:5484] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 8 to 9, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:20.187371 20789 consensus_queue.cc:1035] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:20.187450 13318 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.190559 21014 consensus_queue.cc:1035] T 1d8443bed0da49d0b24511a8468c2373 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:20.209462 16151 consensus_queue.cc:237] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:20.216665 13318 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:15:20.217083 13187 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240502 14:15:20.217990 21026 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:15:20.220818 18885 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.221510 13185 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.222211 20920 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:15:20.223807 13319 raft_consensus.cc:2932] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.227121 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0e393a7736b14d5483973f9cb89058f8 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:20.232424 13038 catalog_manager.cc:5484] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 9 to 10, VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) evicted. New cstate: current_term: 2 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.238176 13559 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e393a7736b14d5483973f9cb89058f8 with delete_type TABLET_DATA_TOMBSTONED (TS 7eb4ecb2379b44c0954f2658c4a04350 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:58526
I20240502 14:15:20.239087 21023 ts_tablet_manager.cc:1888] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:15:20.240358 21023 log.cc:1196] T 0854e9f4851e400f9af18181a71ae8f7 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/0854e9f4851e400f9af18181a71ae8f7
I20240502 14:15:20.241323 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 0854e9f4851e400f9af18181a71ae8f7 (table rebalance_test_table_19 [id=3da4fce4919b44df830cd0c2062a894e]) successfully deleted
I20240502 14:15:20.249537 21023 tablet_replica.cc:330] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350: stopping tablet replica
I20240502 14:15:20.249979 21023 raft_consensus.cc:2229] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:20.250346 21023 raft_consensus.cc:2258] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:20.251488 21023 ts_tablet_manager.cc:1875] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:20.267602 21023 ts_tablet_manager.cc:1888] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240502 14:15:20.267776 21023 log.cc:1196] T 0e393a7736b14d5483973f9cb89058f8 P 7eb4ecb2379b44c0954f2658c4a04350: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/wals/0e393a7736b14d5483973f9cb89058f8
I20240502 14:15:20.268623 13029 catalog_manager.cc:4830] TS 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): tablet 0e393a7736b14d5483973f9cb89058f8 (table rebalance_test_table_9 [id=90f309e38a4949de93b82598a1e7e310]) successfully deleted
I20240502 14:15:20.277124 13318 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "76c85c912ab74c32bfed536ea38d23f6"
dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59686
I20240502 14:15:20.277304 13318 raft_consensus.cc:595] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:20.282760 21022 raft_consensus.cc:979] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d: : Instructing follower 8b47d4390f8343a493d6c00123785968 to start an election
I20240502 14:15:20.282877 21022 raft_consensus.cc:1067] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Signalling peer 8b47d4390f8343a493d6c00123785968 to start an election
I20240502 14:15:20.283411 16419 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "76c85c912ab74c32bfed536ea38d23f6"
dest_uuid: "8b47d4390f8343a493d6c00123785968"
from {username='slave'} at 127.6.196.194:47545
I20240502 14:15:20.283578 16419 raft_consensus.cc:482] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:20.283691 16419 raft_consensus.cc:3037] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:20.286181 16419 raft_consensus.cc:504] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:20.288687 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c85c912ab74c32bfed536ea38d23f6" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:20.288864 16152 raft_consensus.cc:3037] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:20.288868 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c85c912ab74c32bfed536ea38d23f6" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:20.289021 13318 raft_consensus.cc:3032] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:20.289098 13318 raft_consensus.cc:729] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:20.289331 13318 consensus_queue.cc:260] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:20.289685 13318 raft_consensus.cc:3037] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:20.288661 16419 leader_election.cc:288] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 2 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:20.291064 16152 raft_consensus.cc:2454] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 2.
I20240502 14:15:20.292239 13318 raft_consensus.cc:2454] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 2.
I20240502 14:15:20.292829 16019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76c85c912ab74c32bfed536ea38d23f6" candidate_uuid: "8b47d4390f8343a493d6c00123785968" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:20.292995 16019 raft_consensus.cc:3037] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:20.293459 16355 leader_election.cc:302] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:20.293687 20992 raft_consensus.cc:2781] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:20.293850 20992 raft_consensus.cc:686] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 2 LEADER]: Becoming Leader. State: Replica: 8b47d4390f8343a493d6c00123785968, State: Running, Role: LEADER
I20240502 14:15:20.294199 20992 consensus_queue.cc:237] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:20.295559 16019 raft_consensus.cc:2454] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b47d4390f8343a493d6c00123785968 in term 2.
I20240502 14:15:20.301609 13038 catalog_manager.cc:5484] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 reported cstate change: term changed from 1 to 2, leader changed from cd67493f6f0142f896010f2b8a39887d (127.6.196.194) to 8b47d4390f8343a493d6c00123785968 (127.6.196.200). New cstate: current_term: 2 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:20.351788 20037 consensus_queue.cc:1035] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:20.375385 20474 consensus_queue.cc:1035] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:20.404248 21014 consensus_queue.cc:1035] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:20.419118 21017 consensus_queue.cc:1035] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:20.422015 20789 raft_consensus.cc:1050] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55: attempting to promote NON_VOTER 8b47d4390f8343a493d6c00123785968 to VOTER
I20240502 14:15:20.423142 20789 consensus_queue.cc:237] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:20.435719 13187 raft_consensus.cc:1261] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:20.436324 20609 consensus_queue.cc:1035] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:20.436549 13445 raft_consensus.cc:1261] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:20.437542 20609 consensus_queue.cc:1035] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:20.441210 16420 raft_consensus.cc:1261] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968 [term 1 LEARNER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:20.444233 20609 consensus_queue.cc:1035] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:20.451303 20609 raft_consensus.cc:2932] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:20.457409 13186 raft_consensus.cc:2932] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:20.464078 13445 raft_consensus.cc:2932] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:20.470731 13039 catalog_manager.cc:5484] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:20.471168 16420 raft_consensus.cc:2932] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8b47d4390f8343a493d6c00123785968 (127.6.196.200) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:20.478564 16019 consensus_queue.cc:237] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:20.480247 16420 raft_consensus.cc:1261] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:20.480481 13445 raft_consensus.cc:1261] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:20.481061 20788 consensus_queue.cc:1035] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:20.481388 20609 consensus_queue.cc:1035] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:20.483603 20788 raft_consensus.cc:2932] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:20.485267 16420 raft_consensus.cc:2932] T fae5b25ec9694cea87381d27f4d46384 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:20.485391 13445 raft_consensus.cc:2932] T fae5b25ec9694cea87381d27f4d46384 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } }
I20240502 14:15:20.486917 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fae5b25ec9694cea87381d27f4d46384 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:20.489519 13038 catalog_manager.cc:5484] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 3 to 4, VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) evicted. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:20.493811 13166 tablet_service.cc:1514] Processing DeleteTablet for tablet fae5b25ec9694cea87381d27f4d46384 with delete_type TABLET_DATA_TOMBSTONED (TS 20a967bad5504b1e9c6fb720ecdd1625 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:37690
I20240502 14:15:20.494660 21078 tablet_replica.cc:330] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625: stopping tablet replica
I20240502 14:15:20.494932 21078 raft_consensus.cc:2229] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:20.495187 21078 raft_consensus.cc:2258] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:20.495882 21078 ts_tablet_manager.cc:1875] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:20.501366 21078 ts_tablet_manager.cc:1888] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:20.501974 21078 log.cc:1196] T fae5b25ec9694cea87381d27f4d46384 P 20a967bad5504b1e9c6fb720ecdd1625: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/wals/fae5b25ec9694cea87381d27f4d46384
I20240502 14:15:20.502578 13029 catalog_manager.cc:4830] TS 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): tablet fae5b25ec9694cea87381d27f4d46384 (table rebalancer_extra_table_40 [id=9327d0e10a5e429f8d0387e95ae533aa]) successfully deleted
I20240502 14:15:20.506071 20474 raft_consensus.cc:1050] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3: attempting to promote NON_VOTER cd67493f6f0142f896010f2b8a39887d to VOTER
I20240502 14:15:20.506649 20474 consensus_queue.cc:237] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:20.508747 16018 raft_consensus.cc:1261] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:20.508780 13318 raft_consensus.cc:1261] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:20.508929 16285 raft_consensus.cc:1261] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:20.509642 20475 consensus_queue.cc:1035] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:20.510067 20475 consensus_queue.cc:1035] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:20.510455 20475 consensus_queue.cc:1035] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:20.547479 20474 raft_consensus.cc:2932] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:20.550750 16018 raft_consensus.cc:2932] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:20.551489 16285 raft_consensus.cc:2932] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:20.558938 13045 catalog_manager.cc:5484] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:20.558136 13318 raft_consensus.cc:2932] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:20.565959 13445 consensus_queue.cc:237] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:20.569334 16288 raft_consensus.cc:1261] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:20.569938 13318 raft_consensus.cc:1261] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:20.569949 21088 consensus_queue.cc:1035] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:20.571013 21088 consensus_queue.cc:1035] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:20.574213 20475 raft_consensus.cc:2932] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:20.578677 20643 raft_consensus.cc:1050] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER d1da7f11f0604f78934354a992bb0f55 to VOTER
I20240502 14:15:20.579406 20643 consensus_queue.cc:237] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:20.582391 13318 raft_consensus.cc:2932] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:20.583334 16288 raft_consensus.cc:2932] T c5652dfa60d84803abbac9201e5874db P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:20.586776 20837 raft_consensus.cc:1050] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 to VOTER
I20240502 14:15:20.587563 20837 consensus_queue.cc:237] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:20.589423 16019 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:20.590029 20929 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:20.592437 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c5652dfa60d84803abbac9201e5874db with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:20.592500 13038 catalog_manager.cc:5484] T c5652dfa60d84803abbac9201e5874db P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New cstate: current_term: 1 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:20.595824 16002 tablet_service.cc:1514] Processing DeleteTablet for tablet c5652dfa60d84803abbac9201e5874db with delete_type TABLET_DATA_TOMBSTONED (TS d1da7f11f0604f78934354a992bb0f55 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:20.597832 16420 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:20.598372 20929 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:20.601152 13445 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:20.601809 20929 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:20.623558 13187 raft_consensus.cc:1261] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:20.624011 21114 tablet_replica.cc:330] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:20.624272 21114 raft_consensus.cc:2229] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:20.624593 21114 raft_consensus.cc:2258] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:20.624730 16420 raft_consensus.cc:1261] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:20.625160 16152 raft_consensus.cc:1261] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:20.625749 21114 ts_tablet_manager.cc:1875] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:20.626024 21011 consensus_queue.cc:1035] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:20.626521 21011 consensus_queue.cc:1035] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:20.626406 20643 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:20.626983 21011 consensus_queue.cc:1035] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:20.628849 13445 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:20.632758 21114 ts_tablet_manager.cc:1888] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:20.632905 21114 log.cc:1196] T c5652dfa60d84803abbac9201e5874db P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/c5652dfa60d84803abbac9201e5874db
I20240502 14:15:20.633731 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet c5652dfa60d84803abbac9201e5874db (table rebalancer_extra_table_30 [id=fc0747d6473d42628aa0af6e2ee96269]) successfully deleted
I20240502 14:15:20.635382 13038 catalog_manager.cc:5484] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:20.638347 16018 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:20.655869 16418 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:20.661751 21022 raft_consensus.cc:2932] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.663380 13578 consensus_queue.cc:237] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:20.664901 16018 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:20.665467 16418 raft_consensus.cc:1261] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:20.665485 20923 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:20.666289 20923 consensus_queue.cc:1035] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:20.667685 20923 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:20.668313 16018 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:20.673527 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 27857198d4ec430ebee897c90141c58d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:20.673889 16419 raft_consensus.cc:2932] T 27857198d4ec430ebee897c90141c58d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:20.675248 13038 catalog_manager.cc:5484] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 8 to 9, VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) evicted. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:20.675534 16421 raft_consensus.cc:2932] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.680430 16018 consensus_queue.cc:237] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:20.687160 13427 tablet_service.cc:1514] Processing DeleteTablet for tablet 27857198d4ec430ebee897c90141c58d with delete_type TABLET_DATA_TOMBSTONED (TS 88b77f6edc1c4a72b548bf47c05704f3 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42656
I20240502 14:15:20.691181 13045 catalog_manager.cc:5484] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.692361 13187 raft_consensus.cc:2932] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.695575 21127 tablet_replica.cc:330] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3: stopping tablet replica
I20240502 14:15:20.696638 21127 raft_consensus.cc:2229] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:20.697235 21127 raft_consensus.cc:2258] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:20.701563 16152 raft_consensus.cc:2932] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.703125 21127 ts_tablet_manager.cc:1875] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:20.705718 16152 raft_consensus.cc:1261] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:20.706166 20788 consensus_queue.cc:1035] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:20.708163 15955 consensus_peers.cc:485] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 -> Peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Couldn't send request to peer 7eb4ecb2379b44c0954f2658c4a04350. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 524a9a50c4e843a1ad20f071c83b5aec. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:20.708670 13187 raft_consensus.cc:1261] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:20.709488 20609 consensus_queue.cc:1035] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:20.709295 21068 raft_consensus.cc:2932] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:20.714040 13318 consensus_queue.cc:237] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:20.714076 13187 raft_consensus.cc:2932] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:20.714442 21127 ts_tablet_manager.cc:1888] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240502 14:15:20.714524 21127 log.cc:1196] T 27857198d4ec430ebee897c90141c58d P 88b77f6edc1c4a72b548bf47c05704f3: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/wals/27857198d4ec430ebee897c90141c58d
I20240502 14:15:20.716295 13185 raft_consensus.cc:1261] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:20.716115 16152 raft_consensus.cc:2932] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:20.718423 13029 catalog_manager.cc:4830] TS 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): tablet 27857198d4ec430ebee897c90141c58d (table rebalance_test_table_1 [id=160fdc60b8f84393830bdcb0e0d4769d]) successfully deleted
I20240502 14:15:20.718755 20837 consensus_queue.cc:1035] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:20.720520 16151 raft_consensus.cc:1261] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:20.721036 21022 consensus_queue.cc:1035] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:20.724344 21022 raft_consensus.cc:2932] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.724783 13045 catalog_manager.cc:5484] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:20.727690 13186 raft_consensus.cc:2932] T cd06e00fd93340099df5e020716033a9 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.727859 16151 raft_consensus.cc:2932] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.731508 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet cd06e00fd93340099df5e020716033a9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:20.735383 13038 catalog_manager.cc:5484] T cd06e00fd93340099df5e020716033a9 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:20.738778 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet cd06e00fd93340099df5e020716033a9 with delete_type TABLET_DATA_TOMBSTONED (TS 8b47d4390f8343a493d6c00123785968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:20.743690 21131 tablet_replica.cc:330] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:20.743933 21131 raft_consensus.cc:2229] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:20.744199 21131 raft_consensus.cc:2258] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:20.745043 21131 ts_tablet_manager.cc:1875] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:20.752303 21131 ts_tablet_manager.cc:1888] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:20.752489 21131 log.cc:1196] T cd06e00fd93340099df5e020716033a9 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/cd06e00fd93340099df5e020716033a9
I20240502 14:15:20.753373 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet cd06e00fd93340099df5e020716033a9 (table rebalancer_extra_table_26 [id=a2fc7359aefc4890a033bb4d687e45ff]) successfully deleted
I20240502 14:15:20.766989 16151 consensus_queue.cc:237] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:20.769611 13318 raft_consensus.cc:1261] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:20.769706 16088 consensus_peers.cc:485] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 990da252c05c45d8b7a3e47288f1ab61. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:20.770047 16420 raft_consensus.cc:1261] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:20.770502 20603 consensus_queue.cc:1035] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:20.770975 21137 consensus_queue.cc:1035] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:20.772925 20603 raft_consensus.cc:2932] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:20.776898 13319 raft_consensus.cc:1261] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:20.777369 16420 raft_consensus.cc:2932] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:20.778515 13319 raft_consensus.cc:2932] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:20.782305 13038 catalog_manager.cc:5484] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:20.784318 20994 consensus_queue.cc:1035] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:20.810010 16018 raft_consensus.cc:1261] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:20.810583 20992 consensus_queue.cc:1035] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:20.811132 16288 consensus_queue.cc:237] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
W20240502 14:15:20.813542 16219 consensus_peers.cc:485] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e95022a368ec47949d1003c39a7fe433. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:20.814030 16152 raft_consensus.cc:1261] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:20.814101 16019 raft_consensus.cc:1261] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:20.814525 20610 consensus_queue.cc:1035] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:20.814765 21017 consensus_queue.cc:1035] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:20.818408 20610 raft_consensus.cc:2932] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:20.823798 16151 raft_consensus.cc:1261] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:20.824237 20994 consensus_queue.cc:1035] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:20.823769 16019 raft_consensus.cc:2932] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:20.824543 16152 raft_consensus.cc:2932] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:20.830734 13038 catalog_manager.cc:5484] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:20.866008 16152 consensus_queue.cc:237] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
W20240502 14:15:20.868551 16088 consensus_peers.cc:485] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba8299c5f722440e92803100e7240723. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:20.869365 16420 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:20.869802 20603 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:20.872395 13318 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240502 14:15:20.872675 20920 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:20.873158 16420 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:20.876466 21137 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240502 14:15:20.879325 13038 catalog_manager.cc:5484] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 7 to 8, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:20.892241 13318 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:21.124987 21243 ts_tablet_manager.cc:902] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350: Initiating tablet copy from peer d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:21.145525 21243 tablet_copy_client.cc:308] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.197:43695
I20240502 14:15:21.146099 16043 tablet_copy_service.cc:140] P d1da7f11f0604f78934354a992bb0f55: Received BeginTabletCopySession request for tablet 524a9a50c4e843a1ad20f071c83b5aec from peer 7eb4ecb2379b44c0954f2658c4a04350 ({username='slave'} at 127.6.196.196:59521)
I20240502 14:15:21.146242 16043 tablet_copy_service.cc:161] P d1da7f11f0604f78934354a992bb0f55: Beginning new tablet copy session on tablet 524a9a50c4e843a1ad20f071c83b5aec from peer 7eb4ecb2379b44c0954f2658c4a04350 at {username='slave'} at 127.6.196.196:59521: session id = 7eb4ecb2379b44c0954f2658c4a04350-524a9a50c4e843a1ad20f071c83b5aec
I20240502 14:15:21.148998 16043 tablet_copy_source_session.cc:205] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:21.150271 21243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524a9a50c4e843a1ad20f071c83b5aec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:21.156596 21243 tablet_copy_client.cc:783] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:21.156806 21243 tablet_copy_client.cc:647] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:21.158396 21243 tablet_copy_client.cc:523] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:21.161412 21248 ts_tablet_manager.cc:902] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:21.161592 21243 tablet_bootstrap.cc:492] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:21.171201 21248 tablet_copy_client.cc:308] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.198:42041
I20240502 14:15:21.172291 16175 tablet_copy_service.cc:140] P c86f7440570f4796babfd44c3d7fd847: Received BeginTabletCopySession request for tablet 990da252c05c45d8b7a3e47288f1ab61 from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:41633)
I20240502 14:15:21.172439 16175 tablet_copy_service.cc:161] P c86f7440570f4796babfd44c3d7fd847: Beginning new tablet copy session on tablet 990da252c05c45d8b7a3e47288f1ab61 from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:41633: session id = 20a967bad5504b1e9c6fb720ecdd1625-990da252c05c45d8b7a3e47288f1ab61
I20240502 14:15:21.174435 21243 tablet_bootstrap.cc:492] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:21.174789 21243 tablet_bootstrap.cc:492] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap complete.
I20240502 14:15:21.174839 16175 tablet_copy_source_session.cc:205] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:21.175027 21243 ts_tablet_manager.cc:1367] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.000s
I20240502 14:15:21.175766 21243 raft_consensus.cc:348] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:21.175987 21243 raft_consensus.cc:729] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: LEARNER
I20240502 14:15:21.175993 21248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 990da252c05c45d8b7a3e47288f1ab61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:21.176182 21243 consensus_queue.cc:260] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:21.178099 21243 ts_tablet_manager.cc:1398] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:21.180770 16043 tablet_copy_service.cc:342] P d1da7f11f0604f78934354a992bb0f55: Request end of tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-524a9a50c4e843a1ad20f071c83b5aec received from {username='slave'} at 127.6.196.196:59521
I20240502 14:15:21.180907 16043 tablet_copy_service.cc:433] P d1da7f11f0604f78934354a992bb0f55: ending tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-524a9a50c4e843a1ad20f071c83b5aec on tablet 524a9a50c4e843a1ad20f071c83b5aec with peer 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:15:21.181766 21248 tablet_copy_client.cc:783] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:21.181991 21248 tablet_copy_client.cc:647] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:21.183557 21248 tablet_copy_client.cc:523] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:21.185892 21248 tablet_bootstrap.cc:492] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:21.190538 21248 tablet_bootstrap.cc:492] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:21.190838 21248 tablet_bootstrap.cc:492] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:15:21.191015 21248 ts_tablet_manager.cc:1367] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 14:15:21.191551 21248 raft_consensus.cc:348] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:21.191704 21248 raft_consensus.cc:729] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:15:21.191895 21248 consensus_queue.cc:260] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:21.192359 21248 ts_tablet_manager.cc:1398] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.001s user 0.000s sys 0.003s
I20240502 14:15:21.192955 16175 tablet_copy_service.cc:342] P c86f7440570f4796babfd44c3d7fd847: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-990da252c05c45d8b7a3e47288f1ab61 received from {username='slave'} at 127.6.196.193:41633
I20240502 14:15:21.193065 16175 tablet_copy_service.cc:433] P c86f7440570f4796babfd44c3d7fd847: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-990da252c05c45d8b7a3e47288f1ab61 on tablet 990da252c05c45d8b7a3e47288f1ab61 with peer 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:15:21.325994 16420 consensus_queue.cc:237] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:21.328233 16018 raft_consensus.cc:1261] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:21.328894 20992 consensus_queue.cc:1035] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:21.329435 16152 raft_consensus.cc:1261] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:21.330332 20992 consensus_queue.cc:1035] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:21.333459 20992 raft_consensus.cc:2932] T 76c85c912ab74c32bfed536ea38d23f6 P 8b47d4390f8343a493d6c00123785968 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:21.337028 16019 raft_consensus.cc:2932] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:21.337548 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 76c85c912ab74c32bfed536ea38d23f6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:21.339251 16152 raft_consensus.cc:2932] T 76c85c912ab74c32bfed536ea38d23f6 P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:21.346524 13045 catalog_manager.cc:5484] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 3 to 5, VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) evicted. New cstate: current_term: 2 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } }
I20240502 14:15:21.354598 13298 tablet_service.cc:1514] Processing DeleteTablet for tablet 76c85c912ab74c32bfed536ea38d23f6 with delete_type TABLET_DATA_TOMBSTONED (TS cd67493f6f0142f896010f2b8a39887d not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:57492
I20240502 14:15:21.378670 21270 tablet_replica.cc:330] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d: stopping tablet replica
I20240502 14:15:21.378949 21270 raft_consensus.cc:2229] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:21.379179 21270 raft_consensus.cc:2258] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:21.380110 21270 ts_tablet_manager.cc:1875] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:21.385251 21248 ts_tablet_manager.cc:902] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:21.386394 21270 ts_tablet_manager.cc:1888] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 14:15:21.386564 21270 log.cc:1196] T 76c85c912ab74c32bfed536ea38d23f6 P cd67493f6f0142f896010f2b8a39887d: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/wals/76c85c912ab74c32bfed536ea38d23f6
I20240502 14:15:21.387352 13031 catalog_manager.cc:4830] TS cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): tablet 76c85c912ab74c32bfed536ea38d23f6 (table rebalancer_extra_table_38 [id=82dbfe603cd54bde8f4e20b802fcca34]) successfully deleted
I20240502 14:15:21.423776 21276 ts_tablet_manager.cc:902] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:21.441318 21248 tablet_copy_client.cc:308] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.198:42041
I20240502 14:15:21.442126 16175 tablet_copy_service.cc:140] P c86f7440570f4796babfd44c3d7fd847: Received BeginTabletCopySession request for tablet ba8299c5f722440e92803100e7240723 from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:41633)
I20240502 14:15:21.442337 16175 tablet_copy_service.cc:161] P c86f7440570f4796babfd44c3d7fd847: Beginning new tablet copy session on tablet ba8299c5f722440e92803100e7240723 from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:41633: session id = 20a967bad5504b1e9c6fb720ecdd1625-ba8299c5f722440e92803100e7240723
I20240502 14:15:21.444355 16175 tablet_copy_source_session.cc:205] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:21.447026 21248 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba8299c5f722440e92803100e7240723. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:21.452142 21248 tablet_copy_client.cc:783] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:21.452344 21248 tablet_copy_client.cc:647] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:21.452544 21276 tablet_copy_client.cc:308] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.199:33857
I20240502 14:15:21.453181 16308 tablet_copy_service.cc:140] P 5297e816f782430d9a01220f0471a882: Received BeginTabletCopySession request for tablet e95022a368ec47949d1003c39a7fe433 from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:55953)
I20240502 14:15:21.453455 16308 tablet_copy_service.cc:161] P 5297e816f782430d9a01220f0471a882: Beginning new tablet copy session on tablet e95022a368ec47949d1003c39a7fe433 from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:55953: session id = 20a967bad5504b1e9c6fb720ecdd1625-e95022a368ec47949d1003c39a7fe433
I20240502 14:15:21.454224 21248 tablet_copy_client.cc:523] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:21.456959 16308 tablet_copy_source_session.cc:205] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:21.457141 21248 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:21.460626 21276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e95022a368ec47949d1003c39a7fe433. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:21.470774 21276 tablet_copy_client.cc:783] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:21.471029 21276 tablet_copy_client.cc:647] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:21.472509 21276 tablet_copy_client.cc:523] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:21.473675 21248 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:21.474021 21248 tablet_bootstrap.cc:492] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:15:21.474249 21248 ts_tablet_manager.cc:1367] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20240502 14:15:21.474895 21248 raft_consensus.cc:348] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:21.475098 21248 raft_consensus.cc:729] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:15:21.475261 21248 consensus_queue.cc:260] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:21.475507 21276 tablet_bootstrap.cc:492] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:21.475768 21248 ts_tablet_manager.cc:1398] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:21.476366 16175 tablet_copy_service.cc:342] P c86f7440570f4796babfd44c3d7fd847: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-ba8299c5f722440e92803100e7240723 received from {username='slave'} at 127.6.196.193:41633
I20240502 14:15:21.476483 16175 tablet_copy_service.cc:433] P c86f7440570f4796babfd44c3d7fd847: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-ba8299c5f722440e92803100e7240723 on tablet ba8299c5f722440e92803100e7240723 with peer 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:15:21.489099 21276 tablet_bootstrap.cc:492] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:21.489435 21276 tablet_bootstrap.cc:492] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:15:21.489629 21276 ts_tablet_manager.cc:1367] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.005s
I20240502 14:15:21.490227 21276 raft_consensus.cc:348] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:21.490411 21276 raft_consensus.cc:729] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:15:21.490574 21276 consensus_queue.cc:260] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:21.491016 21276 ts_tablet_manager.cc:1398] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:21.492805 16308 tablet_copy_service.cc:342] P 5297e816f782430d9a01220f0471a882: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-e95022a368ec47949d1003c39a7fe433 received from {username='slave'} at 127.6.196.193:55953
I20240502 14:15:21.492924 16308 tablet_copy_service.cc:433] P 5297e816f782430d9a01220f0471a882: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-e95022a368ec47949d1003c39a7fe433 on tablet e95022a368ec47949d1003c39a7fe433 with peer 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:15:21.622694 13578 raft_consensus.cc:1203] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:21.740901 13185 raft_consensus.cc:1203] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:21.876376 13185 raft_consensus.cc:1203] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240502 14:15:21.898438 13185 raft_consensus.cc:1203] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.7->[1.8-1.8] Dedup: 1.8->[]
I20240502 14:15:22.095521 21313 raft_consensus.cc:1050] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55: attempting to promote NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 to VOTER
I20240502 14:15:22.096164 21313 consensus_queue.cc:237] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:22.098538 13185 raft_consensus.cc:1261] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:22.098717 13578 raft_consensus.cc:1261] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:22.099004 16151 raft_consensus.cc:1261] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:22.099398 21068 consensus_queue.cc:1035] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:22.099781 20789 consensus_queue.cc:1035] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:22.100216 20789 consensus_queue.cc:1035] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:22.113647 21313 raft_consensus.cc:2932] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:22.114322 16151 raft_consensus.cc:2932] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:22.118058 13185 raft_consensus.cc:2932] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:22.120442 13045 catalog_manager.cc:5484] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 reported cstate change: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:22.125109 13578 raft_consensus.cc:2932] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:22.238471 16019 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "524a9a50c4e843a1ad20f071c83b5aec"
dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37358
I20240502 14:15:22.238668 16019 raft_consensus.cc:595] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:22.352726 20920 raft_consensus.cc:1050] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847: attempting to promote NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 to VOTER
I20240502 14:15:22.353469 20920 consensus_queue.cc:237] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:22.355933 13318 raft_consensus.cc:1261] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:22.356590 16420 raft_consensus.cc:1261] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:22.356710 21137 consensus_queue.cc:1035] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:22.359871 20603 consensus_queue.cc:1035] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:22.359989 13186 raft_consensus.cc:1261] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:22.360548 20603 consensus_queue.cc:1035] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:22.366550 20603 raft_consensus.cc:2932] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.372134 13319 raft_consensus.cc:2932] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.376830 21314 raft_consensus.cc:1050] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882: attempting to promote NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 to VOTER
I20240502 14:15:22.376482 13186 raft_consensus.cc:2932] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.377622 21314 consensus_queue.cc:237] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:22.379051 16418 raft_consensus.cc:2932] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.382587 13039 catalog_manager.cc:5484] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.390136 16152 consensus_queue.cc:237] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:22.392016 16152 raft_consensus.cc:1261] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:22.392522 16018 raft_consensus.cc:1261] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:22.393129 21017 consensus_queue.cc:1035] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:22.393729 21017 consensus_queue.cc:1035] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:22.395359 13319 raft_consensus.cc:1261] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:22.395530 13185 raft_consensus.cc:1261] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:22.395992 20920 consensus_queue.cc:1035] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:22.399143 21392 consensus_queue.cc:1035] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:22.403414 13186 raft_consensus.cc:1261] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:22.403859 21335 consensus_queue.cc:1035] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:22.406092 20603 raft_consensus.cc:2932] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.409996 13319 raft_consensus.cc:2932] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.411921 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 990da252c05c45d8b7a3e47288f1ab61 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:22.413754 13187 raft_consensus.cc:2932] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.420383 13038 catalog_manager.cc:5484] T 990da252c05c45d8b7a3e47288f1ab61 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 3 to 4, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:22.427852 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet 990da252c05c45d8b7a3e47288f1ab61 with delete_type TABLET_DATA_TOMBSTONED (TS 8b47d4390f8343a493d6c00123785968 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:22.429829 21402 tablet_replica.cc:330] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:22.430032 21402 raft_consensus.cc:2229] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:22.430245 21402 raft_consensus.cc:2258] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:22.431452 21402 ts_tablet_manager.cc:1875] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:22.433575 21335 raft_consensus.cc:2932] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.434082 13185 raft_consensus.cc:2932] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.436700 16152 raft_consensus.cc:2932] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.439720 13039 catalog_manager.cc:5484] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:22.440137 21402 ts_tablet_manager.cc:1888] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:22.440302 21402 log.cc:1196] T 990da252c05c45d8b7a3e47288f1ab61 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/990da252c05c45d8b7a3e47288f1ab61
I20240502 14:15:22.440507 16019 raft_consensus.cc:2932] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.441211 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet 990da252c05c45d8b7a3e47288f1ab61 (table rebalancer_extra_table_18 [id=9dc6b07ff15645ab9222cc55c5a4a00b]) successfully deleted
I20240502 14:15:22.444236 16288 consensus_queue.cc:237] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:22.446426 16152 raft_consensus.cc:1261] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:22.446858 21335 consensus_queue.cc:1035] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:22.449088 13185 raft_consensus.cc:1261] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:22.448940 21314 raft_consensus.cc:2932] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.449417 16151 raft_consensus.cc:2932] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.451766 21403 consensus_queue.cc:1035] T e95022a368ec47949d1003c39a7fe433 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:22.453289 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e95022a368ec47949d1003c39a7fe433 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:22.453263 13185 raft_consensus.cc:2932] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.458220 13043 catalog_manager.cc:5484] T e95022a368ec47949d1003c39a7fe433 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 3 to 4, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.462772 16002 tablet_service.cc:1514] Processing DeleteTablet for tablet e95022a368ec47949d1003c39a7fe433 with delete_type TABLET_DATA_TOMBSTONED (TS d1da7f11f0604f78934354a992bb0f55 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:22.463527 21406 tablet_replica.cc:330] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:22.463851 21406 raft_consensus.cc:2229] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:22.464150 21406 raft_consensus.cc:2258] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:22.464879 21406 ts_tablet_manager.cc:1875] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:22.470659 21406 ts_tablet_manager.cc:1888] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240502 14:15:22.470768 21406 log.cc:1196] T e95022a368ec47949d1003c39a7fe433 P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/e95022a368ec47949d1003c39a7fe433
I20240502 14:15:22.471457 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet e95022a368ec47949d1003c39a7fe433 (table rebalancer_extra_table_24 [id=ab60746d83874ba487a87d72e6092c84]) successfully deleted
I20240502 14:15:22.496695 20920 raft_consensus.cc:1050] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847: attempting to promote NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 to VOTER
I20240502 14:15:22.497340 20920 consensus_queue.cc:237] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:22.499562 16418 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:22.499665 13319 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:22.500136 21137 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:22.501175 13185 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240502 14:15:22.502070 21137 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:22.502327 21392 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240502 14:15:22.506518 21392 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.507051 13319 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.513506 16418 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.517689 13186 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.517908 13043 catalog_manager.cc:5484] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 8 to 9, 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.522617 16019 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "524a9a50c4e843a1ad20f071c83b5aec"
dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:37380
I20240502 14:15:22.522809 16019 raft_consensus.cc:595] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Received request to transfer leadership
I20240502 14:15:22.534214 16152 consensus_queue.cc:237] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:22.536581 13318 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:15:22.537420 20920 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:15:22.538975 13186 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240502 14:15:22.540148 20920 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240502 14:15:22.541158 21392 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.541760 13319 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.544317 13030 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba8299c5f722440e92803100e7240723 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:22.544884 13186 raft_consensus.cc:2932] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } } }
I20240502 14:15:22.546844 13038 catalog_manager.cc:5484] T ba8299c5f722440e92803100e7240723 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 9 to 10, VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) evicted. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:22.552029 16401 tablet_service.cc:1514] Processing DeleteTablet for tablet ba8299c5f722440e92803100e7240723 with delete_type TABLET_DATA_TOMBSTONED (TS 8b47d4390f8343a493d6c00123785968 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:53312
I20240502 14:15:22.552448 21402 tablet_replica.cc:330] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: stopping tablet replica
I20240502 14:15:22.552660 21402 raft_consensus.cc:2229] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:22.552917 21402 raft_consensus.cc:2258] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:22.554459 21402 ts_tablet_manager.cc:1875] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:22.564574 21313 raft_consensus.cc:979] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55: : Instructing follower 7eb4ecb2379b44c0954f2658c4a04350 to start an election
I20240502 14:15:22.564742 21313 raft_consensus.cc:1067] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Signalling peer 7eb4ecb2379b44c0954f2658c4a04350 to start an election
I20240502 14:15:22.565929 13578 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "524a9a50c4e843a1ad20f071c83b5aec"
dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
from {username='slave'} at 127.6.196.197:40701
I20240502 14:15:22.566169 13578 raft_consensus.cc:482] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:22.566269 13578 raft_consensus.cc:3037] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:22.567232 21402 ts_tablet_manager.cc:1888] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240502 14:15:22.567406 21402 log.cc:1196] T ba8299c5f722440e92803100e7240723 P 8b47d4390f8343a493d6c00123785968: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/wals/ba8299c5f722440e92803100e7240723
I20240502 14:15:22.568073 13030 catalog_manager.cc:4830] TS 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): tablet ba8299c5f722440e92803100e7240723 (table rebalancer_extra_table_6 [id=04fcf0e53404457b985be305407821cb]) successfully deleted
I20240502 14:15:22.569976 13578 raft_consensus.cc:504] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:22.571099 16019 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524a9a50c4e843a1ad20f071c83b5aec" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:22.571414 16019 raft_consensus.cc:3032] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:22.571532 16019 raft_consensus.cc:729] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:22.571592 16152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524a9a50c4e843a1ad20f071c83b5aec" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c86f7440570f4796babfd44c3d7fd847"
I20240502 14:15:22.571820 16152 raft_consensus.cc:3037] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:22.571898 16019 consensus_queue.cc:260] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:22.572402 16019 raft_consensus.cc:3037] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:22.574637 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524a9a50c4e843a1ad20f071c83b5aec" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:22.574887 13186 raft_consensus.cc:3037] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:22.574357 16019 raft_consensus.cc:2454] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 2.
I20240502 14:15:22.577170 13186 raft_consensus.cc:2454] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 2.
I20240502 14:15:22.582489 13514 leader_election.cc:302] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:22.582976 16152 raft_consensus.cc:2454] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 2.
I20240502 14:15:22.585053 13578 leader_election.cc:288] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 election: Requested vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:22.586632 21447 raft_consensus.cc:2781] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:22.586781 21447 raft_consensus.cc:686] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:15:22.587173 21447 consensus_queue.cc:237] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:22.596555 13039 catalog_manager.cc:5484] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 1 to 2, leader changed from d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 2 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:22.964160 16018 raft_consensus.cc:1261] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:22.964630 21444 consensus_queue.cc:1035] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:23.031810 16152 raft_consensus.cc:1261] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:23.032382 21444 consensus_queue.cc:1035] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:23.039744 13186 raft_consensus.cc:1261] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:23.040382 21444 consensus_queue.cc:1035] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:23.524744 13581 consensus_queue.cc:237] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:23.526603 13186 raft_consensus.cc:1261] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:23.526608 16151 raft_consensus.cc:1261] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:23.527045 21444 consensus_queue.cc:1035] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:23.527761 21444 consensus_queue.cc:1035] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:23.529522 21255 raft_consensus.cc:2932] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:23.530243 16151 raft_consensus.cc:2932] T 524a9a50c4e843a1ad20f071c83b5aec P c86f7440570f4796babfd44c3d7fd847 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:23.535308 13045 catalog_manager.cc:5484] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index 3 to 5, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New cstate: current_term: 2 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:23.536990 13029 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 524a9a50c4e843a1ad20f071c83b5aec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:23.537685 13186 raft_consensus.cc:2932] T 524a9a50c4e843a1ad20f071c83b5aec P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER d1da7f11f0604f78934354a992bb0f55 (127.6.196.197) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:23.545214 16002 tablet_service.cc:1514] Processing DeleteTablet for tablet 524a9a50c4e843a1ad20f071c83b5aec with delete_type TABLET_DATA_TOMBSTONED (TS d1da7f11f0604f78934354a992bb0f55 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:41222
I20240502 14:15:23.545842 21486 tablet_replica.cc:330] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55: stopping tablet replica
I20240502 14:15:23.546075 21486 raft_consensus.cc:2229] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240502 14:15:23.546330 21486 raft_consensus.cc:2258] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240502 14:15:23.547029 21486 ts_tablet_manager.cc:1875] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240502 14:15:23.551827 21486 ts_tablet_manager.cc:1888] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240502 14:15:23.551927 21486 log.cc:1196] T 524a9a50c4e843a1ad20f071c83b5aec P d1da7f11f0604f78934354a992bb0f55: Deleting WAL directory at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/wals/524a9a50c4e843a1ad20f071c83b5aec
I20240502 14:15:23.552695 13031 catalog_manager.cc:4830] TS d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695): tablet 524a9a50c4e843a1ad20f071c83b5aec (table rebalancer_extra_table_40 [id=9327d0e10a5e429f8d0387e95ae533aa]) successfully deleted
I20240502 14:15:23.956187 16265 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:23.957300 13559 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240502 14:15:23.989904 13166 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:15:24.011396 13427 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:15:24.025833 16401 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:15:24.149449 16133 tablet_service.cc:1429] Tablet server has 15 leaders and 0 scanners
I20240502 14:15:24.174101 16002 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240502 14:15:24.206110 13298 tablet_service.cc:1429] Tablet server has 19 leaders and 0 scanners
I20240502 14:15:24.359117 16288 consensus_queue.cc:237] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:24.361306 13578 raft_consensus.cc:1261] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.361358 16418 raft_consensus.cc:1261] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:24.361726 16221 consensus_peers.cc:485] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57b8b76eff294d42a45e3c9240ff23c8. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.361898 21335 consensus_queue.cc:1035] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.362530 21335 consensus_queue.cc:1035] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.364454 21335 raft_consensus.cc:2932] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:24.366106 16418 raft_consensus.cc:2932] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:24.368189 13578 raft_consensus.cc:2932] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:24.373800 13039 catalog_manager.cc:5484] T 57b8b76eff294d42a45e3c9240ff23c8 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:24.394203 13319 consensus_queue.cc:237] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:24.396790 13185 raft_consensus.cc:1261] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.397204 16285 raft_consensus.cc:1261] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.397356 20837 consensus_queue.cc:1035] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.397859 21022 consensus_queue.cc:1035] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:24.400360 13251 consensus_peers.cc:485] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d -> Peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Couldn't send request to peer 7eb4ecb2379b44c0954f2658c4a04350. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 939a92466f61423187582d84ab10c0f6. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.406587 21022 raft_consensus.cc:2932] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:24.416568 13040 catalog_manager.cc:5484] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:24.418118 13185 raft_consensus.cc:2932] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:24.420971 16285 raft_consensus.cc:2932] T 939a92466f61423187582d84ab10c0f6 P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:24.439484 16285 consensus_queue.cc:237] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:24.441579 13578 raft_consensus.cc:1261] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:24.441753 16219 consensus_peers.cc:485] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61c065054cdb456da06172650abf0fae. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.441869 16151 raft_consensus.cc:1261] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.442092 21505 consensus_queue.cc:1035] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.442404 21335 consensus_queue.cc:1035] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.445443 21505 raft_consensus.cc:2932] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:24.445919 13578 raft_consensus.cc:2932] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:24.449394 16151 raft_consensus.cc:2932] T 61c065054cdb456da06172650abf0fae P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } } }
I20240502 14:15:24.451666 13046 catalog_manager.cc:5484] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882 reported cstate change: config changed from index -1 to 2, NON_VOTER 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193) added. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:24.471153 13578 consensus_queue.cc:237] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
W20240502 14:15:24.473778 13514 consensus_peers.cc:485] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ed5f3b784fa44b09974bc9caf17049af. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.473871 16019 raft_consensus.cc:1261] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.474376 16420 raft_consensus.cc:1261] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.474395 21584 consensus_queue.cc:1035] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.474820 21255 consensus_queue.cc:1035] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.478327 21255 raft_consensus.cc:2932] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:24.478976 16019 raft_consensus.cc:2932] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:24.478802 16420 raft_consensus.cc:2932] T ed5f3b784fa44b09974bc9caf17049af P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } } }
I20240502 14:15:24.489507 13040 catalog_manager.cc:5484] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: config changed from index -1 to 2, NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:24.515458 13578 consensus_queue.cc:237] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:24.518677 16420 raft_consensus.cc:1261] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:24.519018 13516 consensus_peers.cc:485] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d1e9c049ffe74f20b569b9ac8f8a7592. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.519518 21584 consensus_queue.cc:1035] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.520725 16019 raft_consensus.cc:1261] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.521318 21584 consensus_queue.cc:1035] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.522287 21255 raft_consensus.cc:2932] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:24.526157 16420 raft_consensus.cc:2932] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:24.526376 16019 raft_consensus.cc:2932] T d1e9c049ffe74f20b569b9ac8f8a7592 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:24.532696 13036 catalog_manager.cc:5484] T d1e9c049ffe74f20b569b9ac8f8a7592 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:24.549425 16288 consensus_queue.cc:237] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:24.552124 13318 raft_consensus.cc:1261] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.552392 16420 raft_consensus.cc:1261] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 5297e816f782430d9a01220f0471a882: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:24.552453 16220 consensus_peers.cc:485] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eafe2272cca9440ba7060f64106a73cb. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.552762 21554 consensus_queue.cc:1035] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.553112 21505 consensus_queue.cc:1035] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.555584 21505 raft_consensus.cc:2932] T eafe2272cca9440ba7060f64106a73cb P 5297e816f782430d9a01220f0471a882 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:24.556620 13319 raft_consensus.cc:2932] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:24.557369 16420 raft_consensus.cc:2932] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:24.564349 13046 catalog_manager.cc:5484] T eafe2272cca9440ba7060f64106a73cb P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) added. New cstate: current_term: 1 leader_uuid: "5297e816f782430d9a01220f0471a882" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } } }
I20240502 14:15:24.581688 13185 consensus_queue.cc:237] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
W20240502 14:15:24.586745 13121 consensus_peers.cc:485] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a2fc48c889b481e9b4d249819a196a0. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.586753 16151 raft_consensus.cc:1261] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:24.587702 21587 consensus_queue.cc:1035] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:24.587814 13578 raft_consensus.cc:1261] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240502 14:15:24.588327 21587 consensus_queue.cc:1035] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240502 14:15:24.602605 21587 raft_consensus.cc:2932] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:24.603587 13578 raft_consensus.cc:2932] T 0a2fc48c889b481e9b4d249819a196a0 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:24.603857 16151 raft_consensus.cc:2932] T 0a2fc48c889b481e9b4d249819a196a0 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:24.610008 13036 catalog_manager.cc:5484] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 6 to 7, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:24.628324 13578 consensus_queue.cc:237] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:24.630311 16420 raft_consensus.cc:1261] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:24.630359 16019 raft_consensus.cc:1261] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:15:24.630769 13516 consensus_peers.cc:485] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b648703b8fc742ecbe1dfd2047d4e68d. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.630811 21255 consensus_queue.cc:1035] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:24.631062 21584 consensus_queue.cc:1035] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:24.633595 21584 raft_consensus.cc:2932] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:24.634212 16420 raft_consensus.cc:2932] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:24.637789 16019 raft_consensus.cc:2932] T b648703b8fc742ecbe1dfd2047d4e68d P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:24.643780 13039 catalog_manager.cc:5484] T b648703b8fc742ecbe1dfd2047d4e68d P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 4 to 5, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:24.666479 13185 consensus_queue.cc:237] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
W20240502 14:15:24.668889 13122 consensus_peers.cc:485] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1b6760f0772449088e93bc1eab5e2ab3. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.669099 16152 raft_consensus.cc:1261] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.669656 13446 raft_consensus.cc:1261] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.669605 21587 consensus_queue.cc:1035] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.670408 21587 consensus_queue.cc:1035] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.672255 21612 raft_consensus.cc:2932] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:24.672981 16152 raft_consensus.cc:2932] T 1b6760f0772449088e93bc1eab5e2ab3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:24.678942 13036 catalog_manager.cc:5484] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:24.679448 13446 raft_consensus.cc:2932] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:24.706251 16420 consensus_queue.cc:237] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:24.708662 16152 raft_consensus.cc:1261] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240502 14:15:24.709254 16353 consensus_peers.cc:485] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 -> Peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Couldn't send request to peer 7eb4ecb2379b44c0954f2658c4a04350. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a1c0b2fdd1d64bac8100d797c86fd8b2. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.709319 13446 raft_consensus.cc:1261] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:24.709741 21469 consensus_queue.cc:1035] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:24.710311 21469 consensus_queue.cc:1035] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:24.712288 21641 raft_consensus.cc:2932] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:24.712936 16152 raft_consensus.cc:2932] T a1c0b2fdd1d64bac8100d797c86fd8b2 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:24.713683 13446 raft_consensus.cc:2932] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } } }
I20240502 14:15:24.719801 13036 catalog_manager.cc:5484] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index 4 to 5, NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) added. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:24.736565 16152 consensus_queue.cc:237] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:24.738950 13446 raft_consensus.cc:1261] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.739063 16019 raft_consensus.cc:1261] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.739472 21635 consensus_queue.cc:1035] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.739717 21137 consensus_queue.cc:1035] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:24.739926 16089 consensus_peers.cc:485] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: df55a71945d34af0affa6eccbdf84dae. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.742431 21483 raft_consensus.cc:2932] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:24.743094 16019 raft_consensus.cc:2932] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:24.747963 13046 catalog_manager.cc:5484] T df55a71945d34af0affa6eccbdf84dae P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:24.753265 13446 raft_consensus.cc:2932] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:24.765583 21658 ts_tablet_manager.cc:902] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d: Initiating tablet copy from peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:24.767582 16420 consensus_queue.cc:237] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:24.769901 16019 raft_consensus.cc:1261] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.770080 13445 raft_consensus.cc:1261] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 8b47d4390f8343a493d6c00123785968: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:24.770231 16355 consensus_peers.cc:485] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e93e561ce7ec4e6fa7e211fd1e4fba56. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.770505 21641 consensus_queue.cc:1035] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.770964 21453 consensus_queue.cc:1035] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.773603 21469 raft_consensus.cc:2932] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:24.774097 16019 raft_consensus.cc:2932] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:24.774259 13445 raft_consensus.cc:2932] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } } }
I20240502 14:15:24.780903 21658 tablet_copy_client.cc:308] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.199:33857
I20240502 14:15:24.783943 16308 tablet_copy_service.cc:140] P 5297e816f782430d9a01220f0471a882: Received BeginTabletCopySession request for tablet 57b8b76eff294d42a45e3c9240ff23c8 from peer cd67493f6f0142f896010f2b8a39887d ({username='slave'} at 127.6.196.194:56211)
I20240502 14:15:24.784086 16308 tablet_copy_service.cc:161] P 5297e816f782430d9a01220f0471a882: Beginning new tablet copy session on tablet 57b8b76eff294d42a45e3c9240ff23c8 from peer cd67493f6f0142f896010f2b8a39887d at {username='slave'} at 127.6.196.194:56211: session id = cd67493f6f0142f896010f2b8a39887d-57b8b76eff294d42a45e3c9240ff23c8
I20240502 14:15:24.784487 13040 catalog_manager.cc:5484] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) added. New cstate: current_term: 1 leader_uuid: "8b47d4390f8343a493d6c00123785968" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:24.786340 16308 tablet_copy_source_session.cc:205] T 57b8b76eff294d42a45e3c9240ff23c8 P 5297e816f782430d9a01220f0471a882: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:24.795428 21658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 57b8b76eff294d42a45e3c9240ff23c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:24.804005 21658 tablet_copy_client.cc:783] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:24.804231 21658 tablet_copy_client.cc:647] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:24.806670 21658 tablet_copy_client.cc:523] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:24.809957 21658 tablet_bootstrap.cc:492] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
I20240502 14:15:24.815801 21658 tablet_bootstrap.cc:492] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:24.816119 21658 tablet_bootstrap.cc:492] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d: Bootstrap complete.
I20240502 14:15:24.816306 21658 ts_tablet_manager.cc:1367] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.002s
I20240502 14:15:24.816728 21658 raft_consensus.cc:348] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:24.816860 21658 raft_consensus.cc:729] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: LEARNER
I20240502 14:15:24.816996 21658 consensus_queue.cc:260] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:24.817540 21658 ts_tablet_manager.cc:1398] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:15:24.818120 16308 tablet_copy_service.cc:342] P 5297e816f782430d9a01220f0471a882: Request end of tablet copy session cd67493f6f0142f896010f2b8a39887d-57b8b76eff294d42a45e3c9240ff23c8 received from {username='slave'} at 127.6.196.194:56211
I20240502 14:15:24.818266 16308 tablet_copy_service.cc:433] P 5297e816f782430d9a01220f0471a882: ending tablet copy session cd67493f6f0142f896010f2b8a39887d-57b8b76eff294d42a45e3c9240ff23c8 on tablet 57b8b76eff294d42a45e3c9240ff23c8 with peer cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:24.826396 16152 consensus_queue.cc:237] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:24.828925 16288 raft_consensus.cc:1261] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:24.829034 16418 raft_consensus.cc:1261] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Refusing update from remote peer c86f7440570f4796babfd44c3d7fd847: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240502 14:15:24.829546 16090 consensus_peers.cc:485] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07012e715ded463db5edbb6019ee69cf. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.829574 21137 consensus_queue.cc:1035] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.829843 21635 consensus_queue.cc:1035] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:24.832811 21635 raft_consensus.cc:2932] T 07012e715ded463db5edbb6019ee69cf P c86f7440570f4796babfd44c3d7fd847 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:24.834281 16418 raft_consensus.cc:2932] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:24.841123 13040 catalog_manager.cc:5484] T 07012e715ded463db5edbb6019ee69cf P 8b47d4390f8343a493d6c00123785968 reported cstate change: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New cstate: current_term: 1 leader_uuid: "c86f7440570f4796babfd44c3d7fd847" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:24.861174 16288 raft_consensus.cc:2932] T 07012e715ded463db5edbb6019ee69cf P 5297e816f782430d9a01220f0471a882 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER cd67493f6f0142f896010f2b8a39887d (127.6.196.194) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } } }
I20240502 14:15:24.877894 21682 ts_tablet_manager.cc:902] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350: Initiating tablet copy from peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:24.886238 16019 consensus_queue.cc:237] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:24.899343 13445 raft_consensus.cc:1261] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:24.901532 16152 raft_consensus.cc:1261] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240502 14:15:24.902619 21358 consensus_queue.cc:1035] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240502 14:15:24.903061 21487 consensus_queue.cc:1035] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:15:24.903685 15956 consensus_peers.cc:485] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:24.921211 21487 raft_consensus.cc:2932] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:24.925920 21690 ts_tablet_manager.cc:902] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625: Initiating tablet copy from peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:24.927390 16152 raft_consensus.cc:2932] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:24.936208 13447 raft_consensus.cc:2932] T 8a98395ed74c4c2eb265a141aec14bd3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:24.936906 21682 tablet_copy_client.cc:308] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.194:33193
I20240502 14:15:24.936519 13040 catalog_manager.cc:5484] T 8a98395ed74c4c2eb265a141aec14bd3 P c86f7440570f4796babfd44c3d7fd847 reported cstate change: config changed from index 4 to 5, NON_VOTER 8b47d4390f8343a493d6c00123785968 (127.6.196.200) added. New cstate: current_term: 1 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } } }
I20240502 14:15:24.941037 13339 tablet_copy_service.cc:140] P cd67493f6f0142f896010f2b8a39887d: Received BeginTabletCopySession request for tablet 939a92466f61423187582d84ab10c0f6 from peer 7eb4ecb2379b44c0954f2658c4a04350 ({username='slave'} at 127.6.196.196:49035)
I20240502 14:15:24.941195 13339 tablet_copy_service.cc:161] P cd67493f6f0142f896010f2b8a39887d: Beginning new tablet copy session on tablet 939a92466f61423187582d84ab10c0f6 from peer 7eb4ecb2379b44c0954f2658c4a04350 at {username='slave'} at 127.6.196.196:49035: session id = 7eb4ecb2379b44c0954f2658c4a04350-939a92466f61423187582d84ab10c0f6
I20240502 14:15:24.944321 13339 tablet_copy_source_session.cc:205] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:24.945892 21682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 939a92466f61423187582d84ab10c0f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:24.955772 21682 tablet_copy_client.cc:783] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:24.956164 21682 tablet_copy_client.cc:647] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:24.958324 21682 tablet_copy_client.cc:523] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:24.961344 21682 tablet_bootstrap.cc:492] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap starting.
I20240502 14:15:24.964278 21690 tablet_copy_client.cc:308] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.199:33857
I20240502 14:15:24.964797 16308 tablet_copy_service.cc:140] P 5297e816f782430d9a01220f0471a882: Received BeginTabletCopySession request for tablet 61c065054cdb456da06172650abf0fae from peer 20a967bad5504b1e9c6fb720ecdd1625 ({username='slave'} at 127.6.196.193:55953)
I20240502 14:15:24.964955 16308 tablet_copy_service.cc:161] P 5297e816f782430d9a01220f0471a882: Beginning new tablet copy session on tablet 61c065054cdb456da06172650abf0fae from peer 20a967bad5504b1e9c6fb720ecdd1625 at {username='slave'} at 127.6.196.193:55953: session id = 20a967bad5504b1e9c6fb720ecdd1625-61c065054cdb456da06172650abf0fae
I20240502 14:15:24.967629 16308 tablet_copy_source_session.cc:205] T 61c065054cdb456da06172650abf0fae P 5297e816f782430d9a01220f0471a882: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:24.969007 21690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61c065054cdb456da06172650abf0fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:24.969370 21682 tablet_bootstrap.cc:492] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:24.969728 21682 tablet_bootstrap.cc:492] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350: Bootstrap complete.
I20240502 14:15:24.969906 21682 ts_tablet_manager.cc:1367] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20240502 14:15:24.970438 21682 raft_consensus.cc:348] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:24.970659 21682 raft_consensus.cc:729] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Initialized, Role: LEARNER
I20240502 14:15:24.970829 21682 consensus_queue.cc:260] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:24.971302 21682 ts_tablet_manager.cc:1398] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:24.973888 13339 tablet_copy_service.cc:342] P cd67493f6f0142f896010f2b8a39887d: Request end of tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-939a92466f61423187582d84ab10c0f6 received from {username='slave'} at 127.6.196.196:49035
I20240502 14:15:24.974011 13339 tablet_copy_service.cc:433] P cd67493f6f0142f896010f2b8a39887d: ending tablet copy session 7eb4ecb2379b44c0954f2658c4a04350-939a92466f61423187582d84ab10c0f6 on tablet 939a92466f61423187582d84ab10c0f6 with peer 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:15:24.976045 21690 tablet_copy_client.cc:783] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:24.976220 21690 tablet_copy_client.cc:647] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:24.977592 21690 tablet_copy_client.cc:523] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:24.979882 21690 tablet_bootstrap.cc:492] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap starting.
I20240502 14:15:24.989008 21690 tablet_bootstrap.cc:492] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:24.989316 21690 tablet_bootstrap.cc:492] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625: Bootstrap complete.
I20240502 14:15:24.989524 21690 ts_tablet_manager.cc:1367] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20240502 14:15:24.989946 21690 raft_consensus.cc:348] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:24.990090 21690 raft_consensus.cc:729] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Initialized, Role: LEARNER
I20240502 14:15:24.990243 21690 consensus_queue.cc:260] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:24.990619 21690 ts_tablet_manager.cc:1398] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:24.991351 16308 tablet_copy_service.cc:342] P 5297e816f782430d9a01220f0471a882: Request end of tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-61c065054cdb456da06172650abf0fae received from {username='slave'} at 127.6.196.193:55953
I20240502 14:15:24.991568 16308 tablet_copy_service.cc:433] P 5297e816f782430d9a01220f0471a882: ending tablet copy session 20a967bad5504b1e9c6fb720ecdd1625-61c065054cdb456da06172650abf0fae on tablet 61c065054cdb456da06172650abf0fae with peer 20a967bad5504b1e9c6fb720ecdd1625
I20240502 14:15:25.039744 21738 ts_tablet_manager.cc:902] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:25.047443 21658 ts_tablet_manager.cc:902] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d: Initiating tablet copy from peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:25.053742 21738 tablet_copy_client.cc:308] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:15:25.054335 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet ed5f3b784fa44b09974bc9caf17049af from peer 88b77f6edc1c4a72b548bf47c05704f3 ({username='slave'} at 127.6.196.195:49767)
I20240502 14:15:25.054466 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet ed5f3b784fa44b09974bc9caf17049af from peer 88b77f6edc1c4a72b548bf47c05704f3 at {username='slave'} at 127.6.196.195:49767: session id = 88b77f6edc1c4a72b548bf47c05704f3-ed5f3b784fa44b09974bc9caf17049af
I20240502 14:15:25.058424 13601 tablet_copy_source_session.cc:205] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:25.062634 21738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed5f3b784fa44b09974bc9caf17049af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:25.068334 21738 tablet_copy_client.cc:783] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:25.068543 21738 tablet_copy_client.cc:647] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:25.069905 21738 tablet_copy_client.cc:523] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240502 14:15:25.072959 21738 tablet_bootstrap.cc:492] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap starting.
I20240502 14:15:25.073840 21752 ts_tablet_manager.cc:902] T d1e9c049ffe74f20b569b9ac8f8a7592 P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:25.077967 21658 tablet_copy_client.cc:308] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.193:36687
I20240502 14:15:25.078562 13208 tablet_copy_service.cc:140] P 20a967bad5504b1e9c6fb720ecdd1625: Received BeginTabletCopySession request for tablet 1b6760f0772449088e93bc1eab5e2ab3 from peer cd67493f6f0142f896010f2b8a39887d ({username='slave'} at 127.6.196.194:43735)
I20240502 14:15:25.078696 13208 tablet_copy_service.cc:161] P 20a967bad5504b1e9c6fb720ecdd1625: Beginning new tablet copy session on tablet 1b6760f0772449088e93bc1eab5e2ab3 from peer cd67493f6f0142f896010f2b8a39887d at {username='slave'} at 127.6.196.194:43735: session id = cd67493f6f0142f896010f2b8a39887d-1b6760f0772449088e93bc1eab5e2ab3
I20240502 14:15:25.081440 13208 tablet_copy_source_session.cc:205] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:25.083281 21658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b6760f0772449088e93bc1eab5e2ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:25.085825 21757 ts_tablet_manager.cc:902] T 0a2fc48c889b481e9b4d249819a196a0 P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
F20240502 14:15:25.086663 21757 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0854e9f4851e400f9af18181a71ae8f7.queue_time_us, tablet-download-pool-0854e9f4851e400f9af18181a71ae8f7.queue_time_us) (tablet-download-pool-0854e9f4851e400f9af18181a71ae8f7.run_cpu_time_us, tablet-download-pool-0854e9f4851e400f9af18181a71ae8f7.run_cpu_time_us) (tablet-download-pool-0854e9f4851e400f9af18181a71ae8f7.run_wall_time_us, tablet-download-pool-0854e9f4851e400f9af18181a71ae8f7.run_wall_time_us) (tablet-download-pool-0a2fc48c889b481e9b4d249819a196a0.queue_time_us, tablet-download-pool-0a2fc48c889b481e9b4d249819a196a0.queue_time_us) (tablet-download-pool-21ac2a9fa2154a0bbbe15aeaf178ea1a.queue_time_us, tablet-download-pool-21ac2a9fa2154a0bbbe15aeaf178ea1a.queue_time_us) (tablet-download-pool-21ac2a9fa2154a0bbbe15aeaf178ea1a.run_cpu_time_us, tablet-download-pool-21ac2a9fa2154a0bbbe15aeaf178ea1a.run_cpu_time_us) (tablet-download-pool-21ac2a9fa2154a0bbbe15aeaf178ea1a.run_wall_time_us, tablet-download-pool-21ac2a9fa2154a0bbbe15aeaf178ea1a.run_wall_time_us) (tablet-download-pool-27857198d4ec430ebee897c90141c58d.queue_time_us, tablet-download-pool-27857198d4ec430ebee897c90141c58d.queue_time_us) (tablet-download-pool-27857198d4ec430ebee897c90141c58d.run_cpu_time_us, tablet-download-pool-27857198d4ec430ebee897c90141c58d.run_cpu_time_us) (tablet-download-pool-27857198d4ec430ebee897c90141c58d.run_wall_time_us, tablet-download-pool-27857198d4ec430ebee897c90141c58d.run_wall_time_us) (tablet-download-pool-abd272455c064ba48726d933d53c83ed.queue_time_us, tablet-download-pool-abd272455c064ba48726d933d53c83ed.queue_time_us) (tablet-download-pool-abd272455c064ba48726d933d53c83ed.run_cpu_time_us, tablet-download-pool-abd272455c064ba48726d933d53c83ed.run_cpu_time_us) (tablet-download-pool-abd272455c064ba48726d933d53c83ed.run_wall_time_us, tablet-download-pool-abd272455c064ba48726d933d53c83ed.run_wall_time_us) (tablet-download-pool-ae45cbb45edf41808686de3418231730.queue_time_us, tablet-download-pool-ae45cbb45edf41808686de3418231730.queue_time_us) (tablet-download-pool-ae45cbb45edf41808686de3418231730.run_cpu_time_us, tablet-download-pool-ae45cbb45edf41808686de3418231730.run_cpu_time_us) (tablet-download-pool-ae45cbb45edf41808686de3418231730.run_wall_time_us, tablet-download-pool-ae45cbb45edf41808686de3418231730.run_wall_time_us) (tablet-download-pool-b2905b7940fb45c49ef8b18943dae256.queue_time_us, tablet-download-pool-b2905b7940fb45c49ef8b18943dae256.queue_time_us) (tablet-download-pool-b2905b7940fb45c49ef8b18943dae256.run_cpu_time_us, tablet-download-pool-b2905b7940fb45c49ef8b18943dae256.run_cpu_time_us) (tablet-download-pool-b2905b7940fb45c49ef8b18943dae256.run_wall_time_us, tablet-download-pool-b2905b7940fb45c49ef8b18943dae256.run_wall_time_us) (tablet-download-pool-b648703b8fc742ecbe1dfd2047d4e68d.queue_time_us, tablet-download-pool-b648703b8fc742ecbe1dfd2047d4e68d.queue_time_us) (tablet-download-pool-b648703b8fc742ecbe1dfd2047d4e68d.run_cpu_time_us, tablet-download-pool-b648703b8fc742ecbe1dfd2047d4e68d.run_cpu_time_us) (tablet-download-pool-b648703b8fc742ecbe1dfd2047d4e68d.run_wall_time_us, tablet-download-pool-b648703b8fc742ecbe1dfd2047d4e68d.run_wall_time_us) (tablet-download-pool-ba519bb3fb534e7fb6106f402e27446d.queue_time_us, tablet-download-pool-ba519bb3fb534e7fb6106f402e27446d.queue_time_us) (tablet-download-pool-ba519bb3fb534e7fb6106f402e27446d.run_cpu_time_us, tablet-download-pool-ba519bb3fb534e7fb6106f402e27446d.run_cpu_time_us) (tablet-download-pool-ba519bb3fb534e7fb6106f402e27446d.run_wall_time_us, tablet-download-pool-ba519bb3fb534e7fb6106f402e27446d.run_wall_time_us) (tablet-download-pool-ba8299c5f722440e92803100e7240723.queue_time_us, tablet-download-pool-ba8299c5f722440e92803100e7240723.queue_time_us) (tablet-download-pool-ba8299c5f722440e92803100e7240723.run_cpu_time_us, tablet-download-pool-ba8299c5f722440e92803100e7240723.run_cpu_time_us) (tablet-download-pool-ba8299c5f722440e92803100e7240723.run_wall_time_us, tablet-download-pool-ba8299c5f722440e92803100e7240723.run_wall_time_us) (tablet-download-pool-c3e9160c46d64fa0a964cc39aec91dfd.queue_time_us, tablet-download-pool-c3e9160c46d64fa0a964cc39aec91dfd.queue_time_us) (tablet-download-pool-c3e9160c46d64fa0a964cc39aec91dfd.run_cpu_time_us, tablet-download-pool-c3e9160c46d64fa0a964cc39aec91dfd.run_cpu_time_us) (tablet-download-pool-c3e9160c46d64fa0a964cc39aec91dfd.run_wall_time_us, tablet-download-pool-c3e9160c46d64fa0a964cc39aec91dfd.run_wall_time_us) (tablet-download-pool-d5975e02205a43d5b93ed8ab51f8a0f8.queue_time_us, tablet-download-pool-d5975e02205a43d5b93ed8ab51f8a0f8.queue_time_us) (tablet-download-pool-d5975e02205a43d5b93ed8ab51f8a0f8.run_cpu_time_us, tablet-download-pool-d5975e02205a43d5b93ed8ab51f8a0f8.run_cpu_time_us) (tablet-download-pool-d5975e02205a43d5b93ed8ab51f8a0f8.run_wall_time_us, tablet-download-pool-d5975e02205a43d5b93ed8ab51f8a0f8.run_wall_time_us) (tablet-download-pool-e3be42ac139741d19dd872ab551bebae.queue_time_us, tablet-download-pool-e3be42ac139741d19dd872ab551bebae.queue_time_us) (tablet-download-pool-e3be42ac139741d19dd872ab551bebae.run_cpu_time_us, tablet-download-pool-e3be42ac139741d19dd872ab551bebae.run_cpu_time_us) (tablet-download-pool-e3be42ac139741d19dd872ab551bebae.run_wall_time_us, tablet-download-pool-e3be42ac139741d19dd872ab551bebae.run_wall_time_us) (tablet-download-pool-eab73d39b1f7455bba7aef5e515805c7.queue_time_us, tablet-download-pool-eab73d39b1f7455bba7aef5e515805c7.queue_time_us) (tablet-download-pool-eab73d39b1f7455bba7aef5e515805c7.run_cpu_time_us, tablet-download-pool-eab73d39b1f7455bba7aef5e515805c7.run_cpu_time_us) (tablet-download-pool-eab73d39b1f7455bba7aef5e515805c7.run_wall_time_us, tablet-download-pool-eab73d39b1f7455bba7aef5e515805c7.run_wall_time_us) (tablet-download-pool-fae5b25ec9694cea87381d27f4d46384.queue_time_us, tablet-download-pool-fae5b25ec9694cea87381d27f4d46384.queue_time_us) (tablet-download-pool-fae5b25ec9694cea87381d27f4d46384.run_cpu_time_us, tablet-download-pool-fae5b25ec9694cea87381d27f4d46384.run_cpu_time_us) (tablet-download-pool-fae5b25ec9694cea87381d27f4d46384.run_wall_time_us, tablet-download-pool-fae5b25ec9694cea87381d27f4d46384.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f97682ccdcd google::LogMessage::Fail() at ??:0
@ 0x7f97682d0b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f97682cc7cc google::LogMessage::Flush() at ??:0
@ 0x7f97682cdf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 14:15:25.112712 21658 tablet_copy_client.cc:783] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:25.112963 21658 tablet_copy_client.cc:647] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:25.114771 21738 tablet_bootstrap.cc:492] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:25.115146 21738 tablet_bootstrap.cc:492] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3: Bootstrap complete.
I20240502 14:15:25.115417 21738 ts_tablet_manager.cc:1367] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.002s
I20240502 14:15:25.116153 21738 raft_consensus.cc:348] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:25.116418 21738 raft_consensus.cc:729] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Initialized, Role: LEARNER
I20240502 14:15:25.116613 21738 consensus_queue.cc:260] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: NON_VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: true } }
I20240502 14:15:25.117173 21738 ts_tablet_manager.cc:1398] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
@ 0x7f976b00d742 kudu::TraceMetrics::InternName() at ??:0
I20240502 14:15:25.120133 13601 tablet_copy_service.cc:342] P 7eb4ecb2379b44c0954f2658c4a04350: Request end of tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-ed5f3b784fa44b09974bc9caf17049af received from {username='slave'} at 127.6.196.195:49767
I20240502 14:15:25.120239 13601 tablet_copy_service.cc:433] P 7eb4ecb2379b44c0954f2658c4a04350: ending tablet copy session 88b77f6edc1c4a72b548bf47c05704f3-ed5f3b784fa44b09974bc9caf17049af on tablet ed5f3b784fa44b09974bc9caf17049af with peer 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:15:25.127601 21752 tablet_copy_client.cc:308] T d1e9c049ffe74f20b569b9ac8f8a7592 P 5297e816f782430d9a01220f0471a882: tablet copy: Beginning tablet copy session from remote peer at address 127.6.196.196:46817
I20240502 14:15:25.128264 13601 tablet_copy_service.cc:140] P 7eb4ecb2379b44c0954f2658c4a04350: Received BeginTabletCopySession request for tablet d1e9c049ffe74f20b569b9ac8f8a7592 from peer 5297e816f782430d9a01220f0471a882 ({username='slave'} at 127.6.196.199:53105)
I20240502 14:15:25.128434 13601 tablet_copy_service.cc:161] P 7eb4ecb2379b44c0954f2658c4a04350: Beginning new tablet copy session on tablet d1e9c049ffe74f20b569b9ac8f8a7592 from peer 5297e816f782430d9a01220f0471a882 at {username='slave'} at 127.6.196.199:53105: session id = 5297e816f782430d9a01220f0471a882-d1e9c049ffe74f20b569b9ac8f8a7592
I20240502 14:15:25.130815 21658 tablet_copy_client.cc:523] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f976afd9f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240502 14:15:25.132413 13601 tablet_copy_source_session.cc:205] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350: Tablet Copy: opened 0 blocks and 1 log segments
I20240502 14:15:25.133970 21764 ts_tablet_manager.cc:902] T df55a71945d34af0affa6eccbdf84dae P 8b47d4390f8343a493d6c00123785968: Initiating tablet copy from peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:25.134722 21752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1e9c049ffe74f20b569b9ac8f8a7592. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f976afd3cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f977a4268de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240502 14:15:25.143903 21658 tablet_bootstrap.cc:492] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d: Bootstrap starting.
@ 0x7f977a43edaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 14:15:25.149422 21752 tablet_copy_client.cc:783] T d1e9c049ffe74f20b569b9ac8f8a7592 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 0 data blocks...
I20240502 14:15:25.149677 21752 tablet_copy_client.cc:647] T d1e9c049ffe74f20b569b9ac8f8a7592 P 5297e816f782430d9a01220f0471a882: tablet copy: Starting download of 1 WAL segments...
I20240502 14:15:25.153216 21752 tablet_copy_client.cc:523] T d1e9c049ffe74f20b569b9ac8f8a7592 P 5297e816f782430d9a01220f0471a882: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f977a569480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f977a5a1a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240502 14:15:25.159927 21752 tablet_bootstrap.cc:492] T d1e9c049ffe74f20b569b9ac8f8a7592 P 5297e816f782430d9a01220f0471a882: Bootstrap starting.
I20240502 14:15:25.164656 21766 ts_tablet_manager.cc:902] T b648703b8fc742ecbe1dfd2047d4e68d P 5297e816f782430d9a01220f0471a882: Initiating tablet copy from peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
F20240502 14:15:25.165383 21766 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0854e9f4851e400f9af18181a71ae8f7.queue_time_us, tablet-download-pool-0854e9f4851e400f9af18181a71ae8f7.queue_time_us) (tablet-download-pool-0854e9f4851e400f9af18181a71ae8f7.run_cpu_time_us, tablet-download-pool-0854e9f4851e400f9af18181a71ae8f7.run_cpu_time_us) (tablet-download-pool-0854e9f4851e400f9af18181a71ae8f7.run_wall_time_us, tablet-download-pool-0854e9f4851e400f9af18181a71ae8f7.run_wall_time_us) (tablet-download-pool-0ec7ae12c17247e9b8c03659435bd4e2.queue_time_us, tablet-download-pool-0ec7ae12c17247e9b8c03659435bd4e2.queue_time_us) (tablet-download-pool-0ec7ae12c17247e9b8c03659435bd4e2.run_cpu_time_us, tablet-download-pool-0ec7ae12c17247e9b8c03659435bd4e2.run_cpu_time_us) (tablet-download-pool-0ec7ae12c17247e9b8c03659435bd4e2.run_wall_time_us, tablet-download-pool-0ec7ae12c17247e9b8c03659435bd4e2.run_wall_time_us) (tablet-download-pool-22705e1aee0947d8996575b93efd3f1f.queue_time_us, tablet-download-pool-22705e1aee0947d8996575b93efd3f1f.queue_time_us) (tablet-download-pool-22705e1aee0947d8996575b93efd3f1f.run_cpu_time_us, tablet-download-pool-22705e1aee0947d8996575b93efd3f1f.run_cpu_time_us) (tablet-download-pool-22705e1aee0947d8996575b93efd3f1f.run_wall_time_us, tablet-download-pool-22705e1aee0947d8996575b93efd3f1f.run_wall_time_us) (tablet-download-pool-2e51feadc94c48e9bcecffecacbda4b9.queue_time_us, tablet-download-pool-2e51feadc94c48e9bcecffecacbda4b9.queue_time_us) (tablet-download-pool-2e51feadc94c48e9bcecffecacbda4b9.run_cpu_time_us, tablet-download-pool-2e51feadc94c48e9bcecffecacbda4b9.run_cpu_time_us) (tablet-download-pool-2e51feadc94c48e9bcecffecacbda4b9.run_wall_time_us, tablet-download-pool-2e51feadc94c48e9bcecffecacbda4b9.run_wall_time_us) (tablet-download-pool-80040c60b029481285c50171bcc5ce35.queue_time_us, tablet-download-pool-80040c60b029481285c50171bcc5ce35.queue_time_us) (tablet-download-pool-80040c60b029481285c50171bcc5ce35.run_cpu_time_us, tablet-download-pool-80040c60b029481285c50171bcc5ce35.run_cpu_time_us) (tablet-download-pool-80040c60b029481285c50171bcc5ce35.run_wall_time_us, tablet-download-pool-80040c60b029481285c50171bcc5ce35.run_wall_time_us) (tablet-download-pool-8a6b71f7e77b4084a8904b1045099472.queue_time_us, tablet-download-pool-8a6b71f7e77b4084a8904b1045099472.queue_time_us) (tablet-download-pool-8a6b71f7e77b4084a8904b1045099472.run_cpu_time_us, tablet-download-pool-8a6b71f7e77b4084a8904b1045099472.run_cpu_time_us) (tablet-download-pool-8a6b71f7e77b4084a8904b1045099472.run_wall_time_us, tablet-download-pool-8a6b71f7e77b4084a8904b1045099472.run_wall_time_us) (tablet-download-pool-9a95b9049a344b98be91b9961f3e5bee.queue_time_us, tablet-download-pool-9a95b9049a344b98be91b9961f3e5bee.queue_time_us) (tablet-download-pool-9a95b9049a344b98be91b9961f3e5bee.run_cpu_time_us, tablet-download-pool-9a95b9049a344b98be91b9961f3e5bee.run_cpu_time_us) (tablet-download-pool-9a95b9049a344b98be91b9961f3e5bee.run_wall_time_us, tablet-download-pool-9a95b9049a344b98be91b9961f3e5bee.run_wall_time_us) (tablet-download-pool-9c5329f38f3343baaf43265990d86938.queue_time_us, tablet-download-pool-9c5329f38f3343baaf43265990d86938.queue_time_us) (tablet-download-pool-9c5329f38f3343baaf43265990d86938.run_cpu_time_us, tablet-download-pool-9c5329f38f3343baaf43265990d86938.run_cpu_time_us) (tablet-download-pool-9c5329f38f3343baaf43265990d86938.run_wall_time_us, tablet-download-pool-9c5329f38f3343baaf43265990d86938.run_wall_time_us) (tablet-download-pool-ab51b049c3674b629b55ed29df5b1fab.queue_time_us, tablet-download-pool-ab51b049c3674b629b55ed29df5b1fab.queue_time_us) (tablet-download-pool-ab51b049c3674b629b55ed29df5b1fab.run_cpu_time_us, tablet-download-pool-ab51b049c3674b629b55ed29df5b1fab.run_cpu_time_us) (tablet-download-pool-ab51b049c3674b629b55ed29df5b1fab.run_wall_time_us, tablet-download-pool-ab51b049c3674b629b55ed29df5b1fab.run_wall_time_us) (tablet-download-pool-b648703b8fc742ecbe1dfd2047d4e68d.queue_time_us, tablet-download-pool-b648703b8fc742ecbe1dfd2047d4e68d.queue_time_us) (tablet-download-pool-bbd12f1c9884497b82c941b7231645bb.queue_time_us, tablet-download-pool-bbd12f1c9884497b82c941b7231645bb.queue_time_us) (tablet-download-pool-bbd12f1c9884497b82c941b7231645bb.run_cpu_time_us, tablet-download-pool-bbd12f1c9884497b82c941b7231645bb.run_cpu_time_us) (tablet-download-pool-bbd12f1c9884497b82c941b7231645bb.run_wall_time_us, tablet-download-pool-bbd12f1c9884497b82c941b7231645bb.run_wall_time_us) (tablet-download-pool-d1e9c049ffe74f20b569b9ac8f8a7592.queue_time_us, tablet-download-pool-d1e9c049ffe74f20b569b9ac8f8a7592.queue_time_us) (tablet-download-pool-d1e9c049ffe74f20b569b9ac8f8a7592.run_cpu_time_us, tablet-download-pool-d1e9c049ffe74f20b569b9ac8f8a7592.run_cpu_time_us) (tablet-download-pool-d1e9c049ffe74f20b569b9ac8f8a7592.run_wall_time_us, tablet-download-pool-d1e9c049ffe74f20b569b9ac8f8a7592.run_wall_time_us) (tablet-download-pool-db2bc203d47c4ca1b3329745a0ff7fe1.queue_time_us, tablet-download-pool-db2bc203d47c4ca1b3329745a0ff7fe1.queue_time_us) (tablet-download-pool-db2bc203d47c4ca1b3329745a0ff7fe1.run_cpu_time_us, tablet-download-pool-db2bc203d47c4ca1b3329745a0ff7fe1.run_cpu_time_us) (tablet-download-pool-db2bc203d47c4ca1b3329745a0ff7fe1.run_wall_time_us, tablet-download-pool-db2bc203d47c4ca1b3329745a0ff7fe1.run_wall_time_us) (tablet-download-pool-dee785c61c73426fb5665c1334ff372b.queue_time_us, tablet-download-pool-dee785c61c73426fb5665c1334ff372b.queue_time_us) (tablet-download-pool-dee785c61c73426fb5665c1334ff372b.run_cpu_time_us, tablet-download-pool-dee785c61c73426fb5665c1334ff372b.run_cpu_time_us) (tablet-download-pool-dee785c61c73426fb5665c1334ff372b.run_wall_time_us, tablet-download-pool-dee785c61c73426fb5665c1334ff372b.run_wall_time_us) (tablet-download-pool-f9c268e5a8e7462ab8e940775c73e591.queue_time_us, tablet-download-pool-f9c268e5a8e7462ab8e940775c73e591.queue_time_us) (tablet-download-pool-f9c268e5a8e7462ab8e940775c73e591.run_cpu_time_us, tablet-download-pool-f9c268e5a8e7462ab8e940775c73e591.run_cpu_time_us) (tablet-download-pool-f9c268e5a8e7462ab8e940775c73e591.run_wall_time_us, tablet-download-pool-f9c268e5a8e7462ab8e940775c73e591.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f977a584981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fafa2d1edcd google::LogMessage::Fail() at ??:0
@ 0x7f977a58470e std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fafa2d22b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fafa2d1e7cc google::LogMessage::Flush() at ??:0
@ 0x7fafa2d1ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240502 14:15:25.185664 21770 ts_tablet_manager.cc:902] T eafe2272cca9440ba7060f64106a73cb P c86f7440570f4796babfd44c3d7fd847: Initiating tablet copy from peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
F20240502 14:15:25.186156 21770 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-0a2fc48c889b481e9b4d249819a196a0.queue_time_us, tablet-download-pool-0a2fc48c889b481e9b4d249819a196a0.queue_time_us) (tablet-download-pool-0a2fc48c889b481e9b4d249819a196a0.run_cpu_time_us, tablet-download-pool-0a2fc48c889b481e9b4d249819a196a0.run_cpu_time_us) (tablet-download-pool-0a2fc48c889b481e9b4d249819a196a0.run_wall_time_us, tablet-download-pool-0a2fc48c889b481e9b4d249819a196a0.run_wall_time_us) (tablet-download-pool-0e393a7736b14d5483973f9cb89058f8.queue_time_us, tablet-download-pool-0e393a7736b14d5483973f9cb89058f8.queue_time_us) (tablet-download-pool-0e393a7736b14d5483973f9cb89058f8.run_cpu_time_us, tablet-download-pool-0e393a7736b14d5483973f9cb89058f8.run_cpu_time_us) (tablet-download-pool-0e393a7736b14d5483973f9cb89058f8.run_wall_time_us, tablet-download-pool-0e393a7736b14d5483973f9cb89058f8.run_wall_time_us) (tablet-download-pool-16c817b864c4425eaa85ac4b5aff59fa.queue_time_us, tablet-download-pool-16c817b864c4425eaa85ac4b5aff59fa.queue_time_us) (tablet-download-pool-16c817b864c4425eaa85ac4b5aff59fa.run_cpu_time_us, tablet-download-pool-16c817b864c4425eaa85ac4b5aff59fa.run_cpu_time_us) (tablet-download-pool-16c817b864c4425eaa85ac4b5aff59fa.run_wall_time_us, tablet-download-pool-16c817b864c4425eaa85ac4b5aff59fa.run_wall_time_us) (tablet-download-pool-16d6faae2add42c5bd4af6018a13b841.queue_time_us, tablet-download-pool-16d6faae2add42c5bd4af6018a13b841.queue_time_us) (tablet-download-pool-16d6faae2add42c5bd4af6018a13b841.run_cpu_time_us, tablet-download-pool-16d6faae2add42c5bd4af6018a13b841.run_cpu_time_us) (tablet-download-pool-16d6faae2add42c5bd4af6018a13b841.run_wall_time_us, tablet-download-pool-16d6faae2add42c5bd4af6018a13b841.run_wall_time_us) (tablet-download-pool-3d39f694c8554638a3b10ebf7cd01350.queue_time_us, tablet-download-pool-3d39f694c8554638a3b10ebf7cd01350.queue_time_us) (tablet-download-pool-3d39f694c8554638a3b10ebf7cd01350.run_cpu_time_us, tablet-download-pool-3d39f694c8554638a3b10ebf7cd01350.run_cpu_time_us) (tablet-download-pool-3d39f694c8554638a3b10ebf7cd01350.run_wall_time_us, tablet-download-pool-3d39f694c8554638a3b10ebf7cd01350.run_wall_time_us) (tablet-download-pool-6395ce4c30eb4644b0a711de3e60b9c0.queue_time_us, tablet-download-pool-6395ce4c30eb4644b0a711de3e60b9c0.queue_time_us) (tablet-download-pool-6395ce4c30eb4644b0a711de3e60b9c0.run_cpu_time_us, tablet-download-pool-6395ce4c30eb4644b0a711de3e60b9c0.run_cpu_time_us) (tablet-download-pool-6395ce4c30eb4644b0a711de3e60b9c0.run_wall_time_us, tablet-download-pool-6395ce4c30eb4644b0a711de3e60b9c0.run_wall_time_us) (tablet-download-pool-652e49046212458280095c016a342f47.queue_time_us, tablet-download-pool-652e49046212458280095c016a342f47.queue_time_us) (tablet-download-pool-652e49046212458280095c016a342f47.run_cpu_time_us, tablet-download-pool-652e49046212458280095c016a342f47.run_cpu_time_us) (tablet-download-pool-652e49046212458280095c016a342f47.run_wall_time_us, tablet-download-pool-652e49046212458280095c016a342f47.run_wall_time_us) (tablet-download-pool-718223dd8fd3432cb174a53325dc784c.queue_time_us, tablet-download-pool-718223dd8fd3432cb174a53325dc784c.queue_time_us) (tablet-download-pool-718223dd8fd3432cb174a53325dc784c.run_cpu_time_us, tablet-download-pool-718223dd8fd3432cb174a53325dc784c.run_cpu_time_us) (tablet-download-pool-718223dd8fd3432cb174a53325dc784c.run_wall_time_us, tablet-download-pool-718223dd8fd3432cb174a53325dc784c.run_wall_time_us) (tablet-download-pool-766f50e1e5a44c8ba211a446bcf86705.queue_time_us, tablet-download-pool-766f50e1e5a44c8ba211a446bcf86705.queue_time_us) (tablet-download-pool-766f50e1e5a44c8ba211a446bcf86705.run_cpu_time_us, tablet-download-pool-766f50e1e5a44c8ba211a446bcf86705.run_cpu_time_us) (tablet-download-pool-766f50e1e5a44c8ba211a446bcf86705.run_wall_time_us, tablet-download-pool-766f50e1e5a44c8ba211a446bcf86705.run_wall_time_us) (tablet-download-pool-76b46f71180f4269bee53f206e9c2665.queue_time_us, tablet-download-pool-76b46f71180f4269bee53f206e9c2665.queue_time_us) (tablet-download-pool-76b46f71180f4269bee53f206e9c2665.run_cpu_time_us, tablet-download-pool-76b46f71180f4269bee53f206e9c2665.run_cpu_time_us) (tablet-download-pool-76b46f71180f4269bee53f206e9c2665.run_wall_time_us, tablet-download-pool-76b46f71180f4269bee53f206e9c2665.run_wall_time_us) (tablet-download-pool-7852539de07744caaf385620c5166861.queue_time_us, tablet-download-pool-7852539de07744caaf385620c5166861.queue_time_us) (tablet-download-pool-7852539de07744caaf385620c5166861.run_cpu_time_us, tablet-download-pool-7852539de07744caaf385620c5166861.run_cpu_time_us) (tablet-download-pool-7852539de07744caaf385620c5166861.run_wall_time_us, tablet-download-pool-7852539de07744caaf385620c5166861.run_wall_time_us) (tablet-download-pool-9ad2eb265e0f469787eb1699e339e0ed.queue_time_us, tablet-download-pool-9ad2eb265e0f469787eb1699e339e0ed.queue_time_us) (tablet-download-pool-9ad2eb265e0f469787eb1699e339e0ed.run_cpu_time_us, tablet-download-pool-9ad2eb265e0f469787eb1699e339e0ed.run_cpu_time_us) (tablet-download-pool-9ad2eb265e0f469787eb1699e339e0ed.run_wall_time_us, tablet-download-pool-9ad2eb265e0f469787eb1699e339e0ed.run_wall_time_us) (tablet-download-pool-e3be42ac139741d19dd872ab551bebae.queue_time_us, tablet-download-pool-e3be42ac139741d19dd872ab551bebae.queue_time_us) (tablet-download-pool-e3be42ac139741d19dd872ab551bebae.run_cpu_time_us, tablet-download-pool-e3be42ac139741d19dd872ab551bebae.run_cpu_time_us) (tablet-download-pool-e3be42ac139741d19dd872ab551bebae.run_wall_time_us, tablet-download-pool-e3be42ac139741d19dd872ab551bebae.run_wall_time_us) (tablet-download-pool-eafe2272cca9440ba7060f64106a73cb.queue_time_us, tablet-download-pool-eafe2272cca9440ba7060f64106a73cb.queue_time_us) (tablet-download-pool-eb472259eca84c31b2ff9f33f7e61177.queue_time_us, tablet-download-pool-eb472259eca84c31b2ff9f33f7e61177.queue_time_us) (tablet-download-pool-eb472259eca84c31b2ff9f33f7e61177.run_cpu_time_us, tablet-download-pool-eb472259eca84c31b2ff9f33f7e61177.run_cpu_time_us) (tablet-download-pool-eb472259eca84c31b2ff9f33f7e61177.run_wall_time_us, tablet-download-pool-eb472259eca84c31b2ff9f33f7e61177.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f87d3962dcd google::LogMessage::Fail() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240502 14:15:25.192440 21658 tablet_bootstrap.cc:492] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240502 14:15:25.192842 21658 tablet_bootstrap.cc:492] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d: Bootstrap complete.
I20240502 14:15:25.193140 21658 ts_tablet_manager.cc:1367] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d: Time spent bootstrapping tablet: real 0.049s user 0.005s sys 0.007s
@ 0x7fafa5a5f742 kudu::TraceMetrics::InternName() at ??:0
I20240502 14:15:25.194811 21658 raft_consensus.cc:348] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:25.197964 21658 raft_consensus.cc:729] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Initialized, Role: LEARNER
I20240502 14:15:25.198266 21658 consensus_queue.cc:260] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
@ 0x7f976afddb75 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 14:15:25.202077 21658 ts_tablet_manager.cc:1398] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240502 14:15:25.203953 13208 tablet_copy_service.cc:342] P 20a967bad5504b1e9c6fb720ecdd1625: Request end of tablet copy session cd67493f6f0142f896010f2b8a39887d-1b6760f0772449088e93bc1eab5e2ab3 received from {username='slave'} at 127.6.196.194:43735
I20240502 14:15:25.204062 13208 tablet_copy_service.cc:433] P 20a967bad5504b1e9c6fb720ecdd1625: ending tablet copy session cd67493f6f0142f896010f2b8a39887d-1b6760f0772449088e93bc1eab5e2ab3 on tablet 1b6760f0772449088e93bc1eab5e2ab3 with peer cd67493f6f0142f896010f2b8a39887d
@ 0x7f976afe119c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fafa5a2bf8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f976afe1019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f87d3966b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f976afbc01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f976e3416db start_thread at ??:0
@ 0x7f87d39627cc google::LogMessage::Flush() at ??:0
@ 0x7fafa5a25cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f976717f71f clone at ??:0
@ 0x7f87d3963f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f87d66a3742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f87d666ff8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7fafb4e788de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f87d6669cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7f87e5abc8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f87e5ad4daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240502 14:15:25.258415 13319 raft_consensus.cc:1203] T 57b8b76eff294d42a45e3c9240ff23c8 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
@ 0x7f87e5bff480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f87e5c37a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f87e5c1a981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fafb4e90daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
@ 0x7f87e5c1a70e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f87d6673b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f87d667719c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f87d6677019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fafb4fbb480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
@ 0x7f87d665201e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f87d99d76db start_thread at ??:0
@ 0x7f87d281571f clone at ??:0
@ 0x7fafb4ff3a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7fafb4fd6981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fafb4fd670e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fafa5a2fb75 kudu::ThreadPool::DispatchThread() at ??:0
I20240502 14:15:25.438860 13447 raft_consensus.cc:1203] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
@ 0x7fafa5a3319c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240502 14:15:25.464529 13581 raft_consensus.cc:1203] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
@ 0x7fafa5a33019 std::_Function_handler<>::_M_invoke() at ??:0
I20240502 14:15:25.478803 13187 raft_consensus.cc:1203] T 61c065054cdb456da06172650abf0fae P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fafa5a0e01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fafa8d936db start_thread at ??:0
@ 0x7fafa1bd171f clone at ??:0
I20240502 14:15:25.644668 13319 raft_consensus.cc:1203] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240502 14:15:25.678175 15956 connection.cc:535] client connection to 127.6.196.200:43857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.678455 15956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.680121 13383 connection.cc:535] client connection to 127.6.196.200:43857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.680310 13383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.686559 13515 connection.cc:535] client connection to 127.6.196.200:43857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.686766 13515 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.695628 13252 connection.cc:535] client connection to 127.6.196.200:43857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.695854 13252 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.696548 13121 connection.cc:535] client connection to 127.6.196.200:43857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.696738 13121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.697958 21839 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:15:25.679550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.196.200:43857 (local address 127.6.196.197:54287)
0502 14:15:25.680049 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0502 14:15:25.680072 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0502 14:15:25.680220 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:15:25.696738 (+ 16518us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":208,"threads_started":1}
W20240502 14:15:25.698158 15956 consensus_peers.cc:485] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.698289 15956 consensus_peers.cc:462] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.698387 15956 consensus_peers.cc:485] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.698475 15956 consensus_peers.cc:485] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.698575 15956 consensus_peers.cc:485] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.698658 15956 consensus_peers.cc:485] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.698756 15956 consensus_peers.cc:485] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.703891 21842 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:15:25.687640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.196.200:43857 (local address 127.6.196.196:52235)
0502 14:15:25.702957 (+ 15317us) negotiation.cc:107] Waiting for socket to connect
0502 14:15:25.703121 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":15190,"threads_started":1}
W20240502 14:15:25.704527 13515 consensus_peers.cc:485] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.704641 13515 consensus_peers.cc:485] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.704718 13515 consensus_peers.cc:485] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.704789 13515 consensus_peers.cc:485] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.706409 13121 consensus_peers.cc:485] T b2905b7940fb45c49ef8b18943dae256 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.706524 13121 consensus_peers.cc:485] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.706622 13121 consensus_peers.cc:462] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:25.707042 13515 connection.cc:535] client connection to 127.6.196.198:42041 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.707394 13121 connection.cc:535] client connection to 127.6.196.198:42041 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.708459 15956 connection.cc:535] client connection to 127.6.196.198:42041 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.709160 13121 consensus_peers.cc:485] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.709275 13121 consensus_peers.cc:485] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.709820 21842 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:15:25.707610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.196.198:42041 (local address 127.6.196.196:51997)
0502 14:15:25.709624 (+ 2014us) negotiation.cc:107] Waiting for socket to connect
0502 14:15:25.709691 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":1927}
W20240502 14:15:25.709928 13515 consensus_peers.cc:485] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.710070 13515 consensus_peers.cc:485] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.710165 13515 consensus_peers.cc:485] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.710198 15956 consensus_peers.cc:485] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.710243 13515 consensus_peers.cc:485] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.710294 15956 consensus_peers.cc:485] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.710317 13515 consensus_peers.cc:485] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.710850 15956 consensus_peers.cc:485] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.713887 15956 consensus_peers.cc:462] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:25.717876 13383 connection.cc:535] client connection to 127.6.196.198:42041 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.718315 13383 consensus_peers.cc:485] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.718381 13383 consensus_peers.cc:485] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.718420 13383 consensus_peers.cc:485] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.718449 13383 consensus_peers.cc:485] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.719126 13383 consensus_peers.cc:485] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.719218 13383 consensus_peers.cc:485] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.721279 13252 connection.cc:535] client connection to 127.6.196.198:42041 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.724370 13252 consensus_peers.cc:485] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.724489 13252 consensus_peers.cc:485] T eab73d39b1f7455bba7aef5e515805c7 P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.724565 13252 consensus_peers.cc:485] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.724644 13252 consensus_peers.cc:485] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.724721 13252 consensus_peers.cc:485] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.724805 13252 consensus_peers.cc:485] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.724879 13252 consensus_peers.cc:485] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.725049 13252 consensus_peers.cc:485] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.725140 13252 consensus_peers.cc:485] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.725212 13252 consensus_peers.cc:485] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.725297 13252 consensus_peers.cc:485] T ccee25274d364700921688f1dbd0c7b3 P cd67493f6f0142f896010f2b8a39887d -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.725364 13252 consensus_peers.cc:485] T 3d39f694c8554638a3b10ebf7cd01350 P cd67493f6f0142f896010f2b8a39887d -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.730978 13121 consensus_peers.cc:485] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.780159 13383 consensus_peers.cc:485] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.787070 15956 consensus_peers.cc:485] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.791275 13515 consensus_peers.cc:485] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.815240 15956 consensus_peers.cc:485] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.855185 13384 connection.cc:535] client connection to 127.6.196.199:33857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.855599 15957 connection.cc:535] client connection to 127.6.196.199:33857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.856088 13253 connection.cc:535] client connection to 127.6.196.199:33857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.857106 13516 connection.cc:535] client connection to 127.6.196.199:33857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.857401 21846 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:15:25.855887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.196.199:33857 (local address 127.6.196.195:51943)
0502 14:15:25.856155 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0502 14:15:25.856173 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 14:15:25.856281 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:15:25.856546 (+ 265us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":163}
W20240502 14:15:25.858659 13384 consensus_peers.cc:485] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.858742 13384 consensus_peers.cc:485] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.858783 13384 consensus_peers.cc:485] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.859886 13384 consensus_peers.cc:485] T 0ec7ae12c17247e9b8c03659435bd4e2 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.860788 13122 connection.cc:535] client connection to 127.6.196.199:33857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:25.864425 15957 consensus_peers.cc:485] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.864534 15957 consensus_peers.cc:485] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.864711 13253 consensus_peers.cc:485] T 9a95b9049a344b98be91b9961f3e5bee P cd67493f6f0142f896010f2b8a39887d -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.864820 13253 consensus_peers.cc:485] T 748564520c0c4ebda80e6f8c7f9acd13 P cd67493f6f0142f896010f2b8a39887d -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.864895 13253 consensus_peers.cc:485] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.864977 13253 consensus_peers.cc:485] T 819e6e83e6064c539cfc73a28c43db5b P cd67493f6f0142f896010f2b8a39887d -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.865125 13122 consensus_peers.cc:485] T f9c268e5a8e7462ab8e940775c73e591 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.865201 13122 consensus_peers.cc:485] T 2e51feadc94c48e9bcecffecacbda4b9 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.865274 13122 consensus_peers.cc:485] T 76b46f71180f4269bee53f206e9c2665 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.870060 13516 consensus_peers.cc:462] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:25.870182 13516 consensus_peers.cc:485] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:25.907390 21811 raft_consensus.cc:1050] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350: attempting to promote NON_VOTER 88b77f6edc1c4a72b548bf47c05704f3 to VOTER
I20240502 14:15:25.908411 21811 consensus_queue.cc:237] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:25.912413 13447 raft_consensus.cc:1261] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 LEARNER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:25.913177 16019 raft_consensus.cc:1261] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:25.913192 21773 consensus_queue.cc:1035] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:15:25.913566 13515 consensus_peers.cc:485] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:25.913729 21811 consensus_queue.cc:1035] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:15:25.930776 13384 consensus_peers.cc:485] T 9c5329f38f3343baaf43265990d86938 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:25.933040 21707 raft_consensus.cc:1050] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d: attempting to promote NON_VOTER 7eb4ecb2379b44c0954f2658c4a04350 to VOTER
I20240502 14:15:25.932956 21255 raft_consensus.cc:2932] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:25.933724 21707 consensus_queue.cc:237] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:25.937322 13187 raft_consensus.cc:1261] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:25.937711 13581 raft_consensus.cc:1261] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 LEARNER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:25.938274 21022 consensus_queue.cc:1035] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:25.939401 21022 consensus_queue.cc:1035] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:15:25.939677 15957 consensus_peers.cc:485] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:25.939702 13253 consensus_peers.cc:485] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:25.947474 21566 raft_consensus.cc:2932] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:25.952996 13447 raft_consensus.cc:2932] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:25.954216 16019 raft_consensus.cc:2932] T ed5f3b784fa44b09974bc9caf17049af P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:25.957769 13187 raft_consensus.cc:2932] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:25.960614 13046 catalog_manager.cc:5484] T ed5f3b784fa44b09974bc9caf17049af P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 2 to 3, 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } }
I20240502 14:15:25.965925 13581 raft_consensus.cc:2932] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:25.980966 13040 catalog_manager.cc:5484] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: config changed from index 2 to 3, 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { replace: true } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
W20240502 14:15:25.987383 13384 consensus_peers.cc:485] T dee785c61c73426fb5665c1334ff372b P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.002311 13318 consensus_queue.cc:237] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.004184 13187 raft_consensus.cc:1261] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:26.005604 21707 consensus_queue.cc:1035] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:26.007228 13581 raft_consensus.cc:1261] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240502 14:15:26.009730 21930 raft_consensus.cc:2932] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:26.012041 13187 raft_consensus.cc:2932] T 939a92466f61423187582d84ab10c0f6 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:26.013113 21930 consensus_queue.cc:1035] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:26.013398 13031 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 939a92466f61423187582d84ab10c0f6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240502 14:15:26.017841 13581 raft_consensus.cc:2932] T 939a92466f61423187582d84ab10c0f6 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } } }
I20240502 14:15:26.018508 13042 catalog_manager.cc:5484] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d reported cstate change: config changed from index 3 to 4, VOTER 5297e816f782430d9a01220f0471a882 (127.6.196.199) evicted. New cstate: current_term: 1 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:26.018889 13384 consensus_peers.cc:485] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:26.025558 13031 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:26.029203 13031 catalog_manager.cc:4573] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 939a92466f61423187582d84ab10c0f6: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:26.049803 13122 consensus_peers.cc:485] T 22705e1aee0947d8996575b93efd3f1f P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:26.049917 13516 consensus_peers.cc:462] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:26.089884 13253 consensus_peers.cc:485] T 8a6b71f7e77b4084a8904b1045099472 P cd67493f6f0142f896010f2b8a39887d -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.133908 21791 raft_consensus.cc:1050] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625: attempting to promote NON_VOTER cd67493f6f0142f896010f2b8a39887d to VOTER
I20240502 14:15:26.134991 21791 consensus_queue.cc:237] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.150847 13318 raft_consensus.cc:1261] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d [term 1 LEARNER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:26.151893 21587 consensus_queue.cc:1035] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:15:26.157630 15956 consensus_peers.cc:462] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:26.162315 13121 consensus_peers.cc:485] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.164141 13447 raft_consensus.cc:1261] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240502 14:15:26.168200 21791 consensus_queue.cc:1035] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240502 14:15:26.173602 13121 consensus_peers.cc:462] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.186834 21830 raft_consensus.cc:2932] T 1b6760f0772449088e93bc1eab5e2ab3 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:26.191865 13445 raft_consensus.cc:2932] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:26.198866 13042 catalog_manager.cc:5484] T 1b6760f0772449088e93bc1eab5e2ab3 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:26.198626 13318 raft_consensus.cc:2932] T 1b6760f0772449088e93bc1eab5e2ab3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, cd67493f6f0142f896010f2b8a39887d (127.6.196.194) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } }
I20240502 14:15:26.236210 21255 raft_consensus.cc:482] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.236471 21255 raft_consensus.cc:504] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.237399 21255 leader_election.cc:288] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:26.239080 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a740ff5e0964efcb1fd7dd54dc4094c" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
W20240502 14:15:26.239956 13515 leader_election.cc:334] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.240787 13516 leader_election.cc:302] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:26.241734 21773 raft_consensus.cc:2726] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:15:26.242271 21566 raft_consensus.cc:482] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.242437 21566 raft_consensus.cc:504] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.243091 21566 leader_election.cc:288] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
W20240502 14:15:26.244705 13253 leader_election.cc:334] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:26.244840 13252 leader_election.cc:334] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.244947 13252 leader_election.cc:302] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.245184 21566 raft_consensus.cc:2726] T 8e69d81920d147a38e2542d22ba8352a P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.253754 21877 raft_consensus.cc:482] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.253906 21877 raft_consensus.cc:504] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:26.254623 21877 leader_election.cc:288] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.255903 21994 raft_consensus.cc:482] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
W20240502 14:15:26.256145 13383 leader_election.cc:334] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.256105 21994 raft_consensus.cc:504] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:26.256557 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df55a71945d34af0affa6eccbdf84dae" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:26.256947 21994 leader_election.cc:288] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:26.257740 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8299c5f722440e92803100e7240723" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
W20240502 14:15:26.258206 13121 leader_election.cc:334] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.258402 13122 leader_election.cc:302] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: c86f7440570f4796babfd44c3d7fd847, cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:26.258543 13384 leader_election.cc:302] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:26.259078 21994 raft_consensus.cc:2726] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.259415 21877 raft_consensus.cc:2726] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.275521 21870 raft_consensus.cc:482] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.275674 21870 raft_consensus.cc:504] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:26.276311 21870 leader_election.cc:288] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:26.277192 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e561ce7ec4e6fa7e211fd1e4fba56" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:26.277366 13447 raft_consensus.cc:2454] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:26.278136 15955 leader_election.cc:302] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:26.278956 21877 raft_consensus.cc:482] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.279071 21877 raft_consensus.cc:504] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:26.279567 21877 leader_election.cc:288] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:26.280112 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e561ce7ec4e6fa7e211fd1e4fba56" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
W20240502 14:15:26.280109 15956 leader_election.cc:334] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.280351 16018 raft_consensus.cc:2454] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:26.280709 13384 leader_election.cc:302] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, d1da7f11f0604f78934354a992bb0f55; no voters:
W20240502 14:15:26.280913 13383 leader_election.cc:334] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.285404 21877 raft_consensus.cc:2781] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.285476 21877 raft_consensus.cc:482] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.285532 21877 raft_consensus.cc:3037] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.288115 21877 raft_consensus.cc:504] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
W20240502 14:15:26.289958 13383 leader_election.cc:334] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.295989 21870 raft_consensus.cc:2781] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.296118 21870 raft_consensus.cc:482] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.296201 21870 raft_consensus.cc:3037] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.299367 21870 raft_consensus.cc:504] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:26.301736 13447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e561ce7ec4e6fa7e211fd1e4fba56" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:26.301937 13447 raft_consensus.cc:2379] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d1da7f11f0604f78934354a992bb0f55 in current term 2: Already voted for candidate 88b77f6edc1c4a72b548bf47c05704f3 in this term.
I20240502 14:15:26.304250 21877 leader_election.cc:288] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:26.307287 21870 leader_election.cc:288] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:26.307953 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e561ce7ec4e6fa7e211fd1e4fba56" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:26.308980 16018 raft_consensus.cc:2379] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 88b77f6edc1c4a72b548bf47c05704f3 in current term 2: Already voted for candidate d1da7f11f0604f78934354a992bb0f55 in this term.
I20240502 14:15:26.309346 13384 leader_election.cc:302] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55
W20240502 14:15:26.309964 15956 leader_election.cc:334] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.310086 15956 leader_election.cc:302] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 88b77f6edc1c4a72b548bf47c05704f3, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.312148 21877 raft_consensus.cc:2726] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.313129 21870 raft_consensus.cc:2726] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.341100 21870 raft_consensus.cc:482] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.341315 21870 raft_consensus.cc:504] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:26.342103 21870 leader_election.cc:288] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:26.343041 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e9160c46d64fa0a964cc39aec91dfd" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
W20240502 14:15:26.343932 15956 leader_election.cc:334] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.344049 15956 leader_election.cc:302] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 8b47d4390f8343a493d6c00123785968, cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:26.344435 21870 raft_consensus.cc:2726] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:15:26.356087 21994 raft_consensus.cc:482] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.356238 21994 raft_consensus.cc:504] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:26.356930 21994 leader_election.cc:288] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:26.357051 21255 raft_consensus.cc:482] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.357259 21255 raft_consensus.cc:504] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.357925 21255 leader_election.cc:288] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:26.358223 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9535cdbc4294429593012b088528c449" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
W20240502 14:15:26.358753 13121 leader_election.cc:334] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.358881 13121 leader_election.cc:302] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: 8b47d4390f8343a493d6c00123785968, d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:26.359547 21994 raft_consensus.cc:2726] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.359881 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f961cf70d8ca482ba331241052833fd3" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
W20240502 14:15:26.360286 13515 leader_election.cc:334] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.361666 13516 leader_election.cc:302] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 8b47d4390f8343a493d6c00123785968, cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:26.362044 21931 raft_consensus.cc:2726] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.367022 21877 raft_consensus.cc:482] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.367201 21877 raft_consensus.cc:504] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:26.368417 21707 raft_consensus.cc:482] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.368578 21707 raft_consensus.cc:504] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:26.368757 21877 leader_election.cc:288] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:26.369243 21707 leader_election.cc:288] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
W20240502 14:15:26.370705 13384 leader_election.cc:334] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:26.370968 13252 leader_election.cc:334] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.372680 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8299c5f722440e92803100e7240723" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:26.372895 13187 raft_consensus.cc:2454] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:26.373368 13251 leader_election.cc:302] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters: c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.373852 21707 raft_consensus.cc:2781] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.373941 21707 raft_consensus.cc:482] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.374027 21707 raft_consensus.cc:3037] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.376519 21707 raft_consensus.cc:504] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
W20240502 14:15:26.386101 13252 leader_election.cc:334] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.386293 21707 leader_election.cc:288] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:26.386723 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba8299c5f722440e92803100e7240723" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 10 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:26.386904 13187 raft_consensus.cc:3037] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.391937 13187 raft_consensus.cc:2454] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 2.
I20240502 14:15:26.392377 13251 leader_election.cc:302] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters: c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.392665 21707 raft_consensus.cc:2781] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.392866 21707 raft_consensus.cc:686] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [term 2 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:26.393129 21707 consensus_queue.cc:237] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:26.398801 13041 catalog_manager.cc:5484] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 1 to 2, leader changed from c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 2 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:26.409380 13383 leader_election.cc:334] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.409494 13383 leader_election.cc:302] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.411355 21877 raft_consensus.cc:2726] T 3ab40fe08f714c99a5961cc2ad0915d3 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.425551 21931 raft_consensus.cc:482] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.425709 21931 raft_consensus.cc:504] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.426288 21931 leader_election.cc:288] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:26.426549 21707 raft_consensus.cc:482] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.426692 21707 raft_consensus.cc:504] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.427305 21707 leader_election.cc:288] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
W20240502 14:15:26.428440 13515 leader_election.cc:334] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:26.429947 13252 leader_election.cc:334] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.442463 21707 raft_consensus.cc:482] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.442618 21707 raft_consensus.cc:504] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:26.443218 21707 leader_election.cc:288] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:26.444773 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e393a7736b14d5483973f9cb89058f8" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
W20240502 14:15:26.446218 13252 leader_election.cc:334] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.446331 13252 leader_election.cc:302] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d; no voters: 20a967bad5504b1e9c6fb720ecdd1625, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.446678 21932 raft_consensus.cc:482] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.446784 21932 raft_consensus.cc:504] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } }
I20240502 14:15:26.447352 21932 leader_election.cc:288] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.447512 21932 raft_consensus.cc:2726] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:15:26.450414 13252 leader_election.cc:334] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:26.450564 13252 leader_election.cc:334] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.450659 13252 leader_election.cc:302] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d; no voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.451004 21932 raft_consensus.cc:2726] T 90354ba8ac7f40a7a7428901eeda26bd P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.455515 13166 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240502 14:15:26.456841 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f961cf70d8ca482ba331241052833fd3" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:26.457064 13581 raft_consensus.cc:2454] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:26.458950 21870 raft_consensus.cc:482] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.459141 21870 raft_consensus.cc:504] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.459880 21870 leader_election.cc:288] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:26.460800 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a740ff5e0964efcb1fd7dd54dc4094c" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:26.461033 13581 raft_consensus.cc:2454] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 2.
I20240502 14:15:26.461735 15955 leader_election.cc:302] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, d1da7f11f0604f78934354a992bb0f55; no voters:
W20240502 14:15:26.462194 15956 leader_election.cc:334] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.462426 21870 raft_consensus.cc:2781] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:15:26.462565 21870 raft_consensus.cc:482] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.462693 21870 raft_consensus.cc:3037] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:26.466606 21870 raft_consensus.cc:504] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.468181 21932 raft_consensus.cc:482] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.468299 21932 raft_consensus.cc:504] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:26.468813 21932 leader_election.cc:288] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
W20240502 14:15:26.469161 15956 leader_election.cc:334] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.469722 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990da252c05c45d8b7a3e47288f1ab61" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:26.469887 21870 leader_election.cc:288] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:26.469923 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a740ff5e0964efcb1fd7dd54dc4094c" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:26.470074 13581 raft_consensus.cc:3037] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:15:26.470270 13516 leader_election.cc:334] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.470306 13251 leader_election.cc:302] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.470371 13516 leader_election.cc:302] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.470705 21931 raft_consensus.cc:2726] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.470858 21707 raft_consensus.cc:2781] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.470930 21707 raft_consensus.cc:482] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.470996 21707 raft_consensus.cc:3037] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.472362 13581 raft_consensus.cc:2454] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 3.
I20240502 14:15:26.473249 15955 leader_election.cc:302] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, d1da7f11f0604f78934354a992bb0f55; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.473752 21870 raft_consensus.cc:2781] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:26.473932 21870 raft_consensus.cc:686] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [term 3 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:26.474205 21870 consensus_queue.cc:237] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.474575 21566 raft_consensus.cc:482] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.474714 21566 raft_consensus.cc:504] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.475260 21566 leader_election.cc:288] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
W20240502 14:15:26.476176 13252 leader_election.cc:334] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.476310 13252 leader_election.cc:302] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d; no voters: 20a967bad5504b1e9c6fb720ecdd1625, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.476665 21930 raft_consensus.cc:2726] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:26.479269 13252 leader_election.cc:334] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
W20240502 14:15:26.483840 13253 leader_election.cc:334] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.483944 13253 leader_election.cc:302] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.484164 21707 raft_consensus.cc:504] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.484473 21932 raft_consensus.cc:2726] T e7b93f8449ef4e6fb0a7e315130da024 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.485167 21707 leader_election.cc:288] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:26.485689 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f961cf70d8ca482ba331241052833fd3" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:26.485848 13581 raft_consensus.cc:3037] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:26.486469 13252 leader_election.cc:334] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.486876 21487 raft_consensus.cc:482] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.486994 21487 raft_consensus.cc:504] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:26.487509 21487 leader_election.cc:288] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.487959 21876 raft_consensus.cc:482] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.488076 21876 raft_consensus.cc:504] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: false } }
I20240502 14:15:26.488214 13581 raft_consensus.cc:2454] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 2.
I20240502 14:15:26.488616 21876 leader_election.cc:288] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
W20240502 14:15:26.490310 13384 leader_election.cc:334] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:26.490526 15956 leader_election.cc:334] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.490584 13251 leader_election.cc:302] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters: 8b47d4390f8343a493d6c00123785968
W20240502 14:15:26.491537 13383 leader_election.cc:334] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.491637 13383 leader_election.cc:302] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.492012 21876 raft_consensus.cc:2726] T db2bc203d47c4ca1b3329745a0ff7fe1 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:26.492172 15957 leader_election.cc:334] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.492363 15957 leader_election.cc:302] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.493027 21869 raft_consensus.cc:2726] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.493911 13044 catalog_manager.cc:5484] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 2 to 3, leader changed from 8b47d4390f8343a493d6c00123785968 (127.6.196.200) to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 3 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:26.495363 21707 raft_consensus.cc:2781] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.495743 21707 raft_consensus.cc:686] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [term 2 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:26.495954 21707 consensus_queue.cc:237] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.511516 21931 raft_consensus.cc:482] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.511746 21931 raft_consensus.cc:504] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:26.512666 21931 leader_election.cc:288] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:26.512850 21931 raft_consensus.cc:482] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.512981 21931 raft_consensus.cc:504] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
W20240502 14:15:26.516203 13515 leader_election.cc:334] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.517570 21994 raft_consensus.cc:482] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.517737 21994 raft_consensus.cc:504] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:26.517918 21931 leader_election.cc:288] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:26.517800 13044 catalog_manager.cc:5484] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 1 to 2, leader changed from 8b47d4390f8343a493d6c00123785968 (127.6.196.200) to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 2 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:26.518397 21994 leader_election.cc:288] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:26.521162 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1b7405d7ff34356b0aa6ecb12b50817" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
W20240502 14:15:26.521359 13516 leader_election.cc:334] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.521489 13516 leader_election.cc:302] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.522102 21255 raft_consensus.cc:2726] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:26.522542 13515 leader_election.cc:334] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.522672 13515 leader_election.cc:302] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.523190 21931 raft_consensus.cc:2726] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.524552 13559 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
W20240502 14:15:26.542573 13121 leader_election.cc:334] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
W20240502 14:15:26.542748 13121 leader_election.cc:334] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.542855 13121 leader_election.cc:302] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.543072 21994 raft_consensus.cc:2726] T eb27b3bb29754d7399b0314ada6b1ba7 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.551092 13298 tablet_service.cc:1429] Tablet server has 21 leaders and 0 scanners
I20240502 14:15:26.551739 13427 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:15:26.555229 21994 raft_consensus.cc:482] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.555363 21994 raft_consensus.cc:504] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:26.555984 21994 leader_election.cc:288] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:26.556640 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2b8c72c7ba4be58f31ac073e41f2a8" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
W20240502 14:15:26.558280 13121 leader_election.cc:334] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.558442 13121 leader_election.cc:302] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: c86f7440570f4796babfd44c3d7fd847, d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:26.558665 21994 raft_consensus.cc:2726] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.559754 16002 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240502 14:15:26.567832 21487 raft_consensus.cc:482] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.568200 21487 raft_consensus.cc:504] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:26.568779 21487 leader_election.cc:288] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
W20240502 14:15:26.570106 15956 leader_election.cc:334] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
W20240502 14:15:26.570535 15957 leader_election.cc:334] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.570677 15957 leader_election.cc:302] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.571027 21870 raft_consensus.cc:2726] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.573639 21876 raft_consensus.cc:482] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.573761 21876 raft_consensus.cc:504] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:26.574322 21876 leader_election.cc:288] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:26.575649 21487 raft_consensus.cc:482] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.575773 21487 raft_consensus.cc:504] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:26.576361 21487 leader_election.cc:288] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.576493 21994 raft_consensus.cc:482] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.576609 21994 raft_consensus.cc:504] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:26.577200 21994 leader_election.cc:288] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:26.578495 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1b7405d7ff34356b0aa6ecb12b50817" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:26.578694 13581 raft_consensus.cc:2454] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:15:26.579008 13120 leader_election.cc:302] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:15:26.579783 21994 raft_consensus.cc:2781] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.579869 21994 raft_consensus.cc:482] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.579954 21994 raft_consensus.cc:3037] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.582868 21994 raft_consensus.cc:504] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:26.583416 21994 leader_election.cc:288] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:26.583889 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1b7405d7ff34356b0aa6ecb12b50817" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:26.584069 13581 raft_consensus.cc:3037] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:26.585628 15957 leader_election.cc:334] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:26.585826 15956 leader_election.cc:334] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.585913 15956 leader_election.cc:302] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.586192 21830 raft_consensus.cc:482] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.586316 21830 raft_consensus.cc:504] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:26.586403 21487 raft_consensus.cc:2726] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.586759 21994 raft_consensus.cc:482] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.586836 21830 leader_election.cc:288] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.586892 21994 raft_consensus.cc:504] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.587414 21994 leader_election.cc:288] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:26.587625 13581 raft_consensus.cc:2454] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 2.
I20240502 14:15:26.587800 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852539de07744caaf385620c5166861" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:26.588038 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990da252c05c45d8b7a3e47288f1ab61" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:26.588191 13120 leader_election.cc:302] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:15:26.588521 13318 raft_consensus.cc:2454] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:15:26.588673 21830 raft_consensus.cc:2781] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.588858 21830 raft_consensus.cc:686] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:15:26.589342 13122 leader_election.cc:302] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:26.589594 21830 consensus_queue.cc:237] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:26.591213 21791 raft_consensus.cc:2781] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.591306 21791 raft_consensus.cc:482] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.591400 21791 raft_consensus.cc:3037] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:26.591483 13121 leader_election.cc:334] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:26.591750 13121 leader_election.cc:334] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:26.593278 13121 leader_election.cc:334] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
W20240502 14:15:26.593416 13121 leader_election.cc:334] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.593538 21994 raft_consensus.cc:482] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.593711 21994 raft_consensus.cc:504] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:26.594020 21791 raft_consensus.cc:504] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:26.594259 21994 leader_election.cc:288] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.594609 21791 leader_election.cc:288] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.594904 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "990da252c05c45d8b7a3e47288f1ab61" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:26.595077 13318 raft_consensus.cc:3037] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.595433 13120 leader_election.cc:302] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: 7eb4ecb2379b44c0954f2658c4a04350, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.595665 21791 raft_consensus.cc:2726] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:15:26.595654 13516 consensus_peers.cc:462] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.596042 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3b25a9e8b049a88bb999d9362e98da" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:26.597571 13318 raft_consensus.cc:2454] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 2.
I20240502 14:15:26.597990 13122 leader_election.cc:302] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:26.598282 21931 raft_consensus.cc:482] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.598327 21791 raft_consensus.cc:2781] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.598420 21931 raft_consensus.cc:504] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.598642 21791 raft_consensus.cc:686] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:15:26.599025 21931 leader_election.cc:288] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.598934 21791 consensus_queue.cc:237] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
W20240502 14:15:26.600592 13515 leader_election.cc:334] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.601066 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852539de07744caaf385620c5166861" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:26.601255 13186 raft_consensus.cc:2454] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 2.
I20240502 14:15:26.603119 13044 catalog_manager.cc:5484] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 1 to 2, leader changed from 8b47d4390f8343a493d6c00123785968 (127.6.196.200) to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 2 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:26.603719 13514 leader_election.cc:302] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters: c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.604082 21255 raft_consensus.cc:2781] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:15:26.604192 21255 raft_consensus.cc:482] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.604103 21931 raft_consensus.cc:482] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.604429 21931 raft_consensus.cc:504] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.604984 21931 leader_election.cc:288] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:26.604288 21255 raft_consensus.cc:3037] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:15:26.605592 13384 leader_election.cc:334] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:26.605736 13383 leader_election.cc:334] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.605799 13383 leader_election.cc:302] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.606122 20475 raft_consensus.cc:2726] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.607496 21255 raft_consensus.cc:504] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.608060 21255 leader_election.cc:288] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 3 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.609771 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7852539de07744caaf385620c5166861" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:26.609967 13186 raft_consensus.cc:3037] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:15:26.610508 13515 leader_election.cc:334] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:26.610653 13515 leader_election.cc:334] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.611477 20475 raft_consensus.cc:482] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.611640 20475 raft_consensus.cc:504] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:26.612545 20475 leader_election.cc:288] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.612650 13186 raft_consensus.cc:2454] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 3.
I20240502 14:15:26.613211 13514 leader_election.cc:302] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters: c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.613740 21931 raft_consensus.cc:482] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.613862 21931 raft_consensus.cc:504] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:26.614306 21931 leader_election.cc:288] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
W20240502 14:15:26.614391 13383 leader_election.cc:334] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.614459 21931 raft_consensus.cc:2781] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:26.614677 13186 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3b25a9e8b049a88bb999d9362e98da" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:26.614877 13186 raft_consensus.cc:2454] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:26.614876 21931 raft_consensus.cc:686] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [term 3 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:15:26.615209 21931 consensus_queue.cc:237] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.615307 13382 leader_election.cc:302] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters: c86f7440570f4796babfd44c3d7fd847
W20240502 14:15:26.615603 13515 leader_election.cc:334] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.615731 20475 raft_consensus.cc:2781] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.615818 20475 raft_consensus.cc:482] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.615900 20475 raft_consensus.cc:3037] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.616884 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6931796f283a4f20b7fec8c3163b8444" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:26.617265 21487 raft_consensus.cc:482] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.617405 21487 raft_consensus.cc:504] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:26.618417 21487 leader_election.cc:288] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:26.618814 13514 leader_election.cc:302] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.619189 21773 raft_consensus.cc:2726] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.619082 20475 raft_consensus.cc:504] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:26.620013 13187 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9535cdbc4294429593012b088528c449" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:26.620234 13187 raft_consensus.cc:2454] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:26.620244 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a3b25a9e8b049a88bb999d9362e98da" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:26.620623 13185 raft_consensus.cc:3037] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:26.621433 15956 leader_election.cc:334] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.621788 20475 leader_election.cc:288] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 election: Requested vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.623135 13185 raft_consensus.cc:2454] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 2.
I20240502 14:15:26.623710 13382 leader_election.cc:302] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 88b77f6edc1c4a72b548bf47c05704f3; no voters:
I20240502 14:15:26.625193 15955 leader_election.cc:302] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.625690 21870 raft_consensus.cc:482] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.625757 20475 raft_consensus.cc:2781] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.625934 20475 raft_consensus.cc:686] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:26.625866 21870 raft_consensus.cc:504] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:26.626137 20475 consensus_queue.cc:237] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:26.626370 21870 leader_election.cc:288] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
W20240502 14:15:26.626395 13383 leader_election.cc:334] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.626513 21870 raft_consensus.cc:2781] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.626579 21870 raft_consensus.cc:482] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.626672 21870 raft_consensus.cc:3037] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.629520 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2b8c72c7ba4be58f31ac073e41f2a8" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:26.629705 13185 raft_consensus.cc:2454] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:26.629684 21870 raft_consensus.cc:504] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:26.630100 15955 leader_election.cc:302] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:26.630348 21869 raft_consensus.cc:2781] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.630460 21869 raft_consensus.cc:482] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.630542 21869 raft_consensus.cc:3037] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.630712 21870 leader_election.cc:288] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:26.631464 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9535cdbc4294429593012b088528c449" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:26.631649 13185 raft_consensus.cc:3037] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.633086 21869 raft_consensus.cc:504] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:26.633631 21869 leader_election.cc:288] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:26.634455 13185 raft_consensus.cc:2454] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 2.
I20240502 14:15:26.634845 15955 leader_election.cc:302] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:26.635289 21869 raft_consensus.cc:2781] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.635520 21869 raft_consensus.cc:686] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [term 2 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:26.635766 21869 consensus_queue.cc:237] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:26.636740 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e2b8c72c7ba4be58f31ac073e41f2a8" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:26.636902 13185 raft_consensus.cc:3037] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.639612 13185 raft_consensus.cc:2454] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 2.
I20240502 14:15:26.640542 13040 catalog_manager.cc:5484] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 1 to 2, leader changed from c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 2 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:26.641356 15955 leader_election.cc:302] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:26.645210 21487 raft_consensus.cc:482] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.645414 21487 raft_consensus.cc:504] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:26.646061 21487 leader_election.cc:288] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:26.646210 21487 raft_consensus.cc:2781] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.646317 21487 raft_consensus.cc:686] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [term 2 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:26.646564 21487 consensus_queue.cc:237] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
W20240502 14:15:26.647409 15956 leader_election.cc:334] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
W20240502 14:15:26.647869 15956 leader_election.cc:334] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.648306 21707 raft_consensus.cc:482] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.648458 21707 raft_consensus.cc:504] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.648811 21876 raft_consensus.cc:482] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.648984 21876 raft_consensus.cc:504] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:26.649168 21707 leader_election.cc:288] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:26.649726 21876 leader_election.cc:288] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:26.649732 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4af5f41674aaa8587158f25e495fb" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:26.652081 13040 catalog_manager.cc:5484] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 2 to 3, leader changed from c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:26.655483 21707 raft_consensus.cc:482] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.655608 21707 raft_consensus.cc:504] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: NON_VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: true } }
I20240502 14:15:26.656119 21707 leader_election.cc:288] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
W20240502 14:15:26.656598 13383 leader_election.cc:334] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
W20240502 14:15:26.656725 13383 leader_election.cc:334] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.656783 13383 leader_election.cc:302] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847
W20240502 14:15:26.657368 13252 leader_election.cc:334] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:26.657614 13253 leader_election.cc:334] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.657728 13253 leader_election.cc:302] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d; no voters: 5297e816f782430d9a01220f0471a882, 88b77f6edc1c4a72b548bf47c05704f3
I20240502 14:15:26.658236 21707 raft_consensus.cc:2726] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:26.658797 13253 leader_election.cc:334] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.658916 13253 leader_election.cc:302] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d; no voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.658973 21876 raft_consensus.cc:2726] T 2fc251d4e82e452d989405e121d08dd4 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.659231 21707 raft_consensus.cc:2726] T eafe2272cca9440ba7060f64106a73cb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:26.661626 15956 leader_election.cc:334] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
W20240502 14:15:26.662073 15956 leader_election.cc:334] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.663414 21707 raft_consensus.cc:482] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.663532 21707 raft_consensus.cc:504] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:26.664108 21707 leader_election.cc:288] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:26.664535 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b34f2a22306459784a1ad7b81cd1683" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
W20240502 14:15:26.664884 13121 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.196.198:42041, user_credentials={real_user=slave}} blocked reactor thread for 66814.2us
W20240502 14:15:26.665442 13252 leader_election.cc:334] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.665995 13516 leader_election.cc:302] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 8b47d4390f8343a493d6c00123785968, cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:26.666287 21931 raft_consensus.cc:2726] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.666733 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e9160c46d64fa0a964cc39aec91dfd" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:26.666944 16018 raft_consensus.cc:2454] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 2.
I20240502 14:15:26.667148 22040 raft_consensus.cc:482] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.667455 21876 raft_consensus.cc:482] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.667291 22040 raft_consensus.cc:504] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:26.667609 21876 raft_consensus.cc:504] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.668236 21876 leader_election.cc:288] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:26.668790 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6931796f283a4f20b7fec8c3163b8444" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:26.668985 13578 raft_consensus.cc:2454] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 1.
I20240502 14:15:26.669318 13120 leader_election.cc:302] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
W20240502 14:15:26.669435 13384 leader_election.cc:334] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.669571 13253 leader_election.cc:302] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d, d1da7f11f0604f78934354a992bb0f55; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.669692 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4af5f41674aaa8587158f25e495fb" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:26.669912 13318 raft_consensus.cc:2454] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 1.
I20240502 14:15:26.670145 21932 raft_consensus.cc:2781] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:15:26.670253 21932 raft_consensus.cc:482] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Starting leader election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.670348 21932 raft_consensus.cc:3037] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:26.675066 21876 raft_consensus.cc:482] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.675199 21876 raft_consensus.cc:504] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: NON_VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: true } }
I20240502 14:15:26.675679 21876 leader_election.cc:288] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:26.676712 13040 catalog_manager.cc:5484] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 1 to 2, leader changed from c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 2 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:26.677700 13384 leader_election.cc:302] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters: 5297e816f782430d9a01220f0471a882
I20240502 14:15:26.677716 22040 leader_election.cc:288] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:26.677816 21791 raft_consensus.cc:2781] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.677899 21791 raft_consensus.cc:482] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.678076 21791 raft_consensus.cc:3037] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.678426 21707 raft_consensus.cc:482] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.678591 21707 raft_consensus.cc:504] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
W20240502 14:15:26.679072 13121 leader_election.cc:334] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
W20240502 14:15:26.679298 13121 leader_election.cc:334] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.679301 21707 leader_election.cc:288] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:26.679400 13121 leader_election.cc:302] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: 88b77f6edc1c4a72b548bf47c05704f3, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.679600 22040 raft_consensus.cc:2726] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.680101 20475 raft_consensus.cc:2781] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.680183 20475 raft_consensus.cc:482] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.680243 20475 raft_consensus.cc:3037] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.682057 13581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b34f2a22306459784a1ad7b81cd1683" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:26.682271 13581 raft_consensus.cc:2454] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
W20240502 14:15:26.682369 13252 leader_election.cc:334] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.682726 20475 raft_consensus.cc:504] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.682922 13251 leader_election.cc:302] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.683219 20475 leader_election.cc:288] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
I20240502 14:15:26.683349 21566 raft_consensus.cc:2781] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.683440 21566 raft_consensus.cc:482] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.683544 21566 raft_consensus.cc:3037] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.686228 21566 raft_consensus.cc:504] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
W20240502 14:15:26.686484 13383 leader_election.cc:334] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
W20240502 14:15:26.686627 13383 leader_election.cc:334] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.686697 13383 leader_election.cc:302] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.687013 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caa4af5f41674aaa8587158f25e495fb" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:26.687120 21876 raft_consensus.cc:2726] T a1c0b2fdd1d64bac8100d797c86fd8b2 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.687175 13318 raft_consensus.cc:3037] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:26.687283 13384 leader_election.cc:334] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.687609 21566 leader_election.cc:288] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
I20240502 14:15:26.688783 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b34f2a22306459784a1ad7b81cd1683" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:26.688956 13578 raft_consensus.cc:3037] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:26.689507 13252 leader_election.cc:334] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.688155 21932 raft_consensus.cc:504] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:26.689677 13318 raft_consensus.cc:2454] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 2.
I20240502 14:15:26.690483 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e9160c46d64fa0a964cc39aec91dfd" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:26.690649 16018 raft_consensus.cc:3037] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:26.691478 13578 raft_consensus.cc:2454] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 2.
I20240502 14:15:26.693523 21932 leader_election.cc:288] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 3 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:26.694770 13251 leader_election.cc:302] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.695915 13384 leader_election.cc:302] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters: 5297e816f782430d9a01220f0471a882
I20240502 14:15:26.696125 20475 raft_consensus.cc:2781] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.696254 20475 raft_consensus.cc:686] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:26.696480 20475 consensus_queue.cc:237] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.696842 16018 raft_consensus.cc:2454] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 3.
W20240502 14:15:26.697244 13252 leader_election.cc:334] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.697419 13253 leader_election.cc:302] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d, d1da7f11f0604f78934354a992bb0f55; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.697238 13044 catalog_manager.cc:5484] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 1 to 2, leader changed from 8b47d4390f8343a493d6c00123785968 (127.6.196.200) to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 2 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:26.698050 21707 raft_consensus.cc:2781] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.698190 21707 raft_consensus.cc:686] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [term 2 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:26.698477 21707 consensus_queue.cc:237] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.699260 21022 raft_consensus.cc:2781] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:26.699409 21022 raft_consensus.cc:686] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [term 3 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:26.699662 21022 consensus_queue.cc:237] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:26.701783 13043 catalog_manager.cc:5484] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 1 to 2, leader changed from 5297e816f782430d9a01220f0471a882 (127.6.196.199) to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 2 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:26.705425 15955 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.196.193:36687, user_credentials={real_user=slave}} blocked reactor thread for 64101.7us
I20240502 14:15:26.705881 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df55a71945d34af0affa6eccbdf84dae" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:26.706049 13445 raft_consensus.cc:2454] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:26.706526 13043 catalog_manager.cc:5484] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 1 to 2, leader changed from c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 2 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:26.708979 15955 leader_election.cc:302] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, d1da7f11f0604f78934354a992bb0f55; no voters: c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.709306 21869 raft_consensus.cc:2781] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.709421 21869 raft_consensus.cc:482] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.709524 21869 raft_consensus.cc:3037] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:26.711005 13121 leader_election.cc:334] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.711640 21791 raft_consensus.cc:504] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:26.711925 22038 raft_consensus.cc:482] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:26.712090 22038 raft_consensus.cc:504] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:26.712262 21869 raft_consensus.cc:504] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:26.712864 22038 leader_election.cc:288] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.713007 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6931796f283a4f20b7fec8c3163b8444" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350"
I20240502 14:15:26.713233 13578 raft_consensus.cc:3037] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.713507 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "df55a71945d34af0affa6eccbdf84dae" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:26.713647 13445 raft_consensus.cc:3037] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:26.715201 15956 leader_election.cc:334] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:26.715386 15956 leader_election.cc:334] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
W20240502 14:15:26.715499 15956 leader_election.cc:334] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.715582 15956 leader_election.cc:302] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.715896 22038 raft_consensus.cc:2726] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240502 14:15:26.712903 21869 leader_election.cc:288] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Requested vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303)
I20240502 14:15:26.715981 13445 raft_consensus.cc:2454] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 2.
I20240502 14:15:26.716301 13578 raft_consensus.cc:2454] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 2.
I20240502 14:15:26.716338 15955 leader_election.cc:302] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, d1da7f11f0604f78934354a992bb0f55; no voters: c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.716794 13120 leader_election.cc:302] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, 7eb4ecb2379b44c0954f2658c4a04350; no voters:
I20240502 14:15:26.717072 21587 raft_consensus.cc:2781] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.717193 21587 raft_consensus.cc:686] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
I20240502 14:15:26.717334 21869 raft_consensus.cc:2781] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.717458 21587 consensus_queue.cc:237] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:26.717698 21869 raft_consensus.cc:686] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [term 2 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:26.717782 21994 raft_consensus.cc:482] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.717906 21994 raft_consensus.cc:504] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:26.717949 21869 consensus_queue.cc:237] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } }
I20240502 14:15:26.718495 21994 leader_election.cc:288] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
W20240502 14:15:26.723764 13122 leader_election.cc:334] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.724131 21791 leader_election.cc:288] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
W20240502 14:15:26.725656 13121 leader_election.cc:334] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:26.726402 13121 leader_election.cc:334] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.726598 13121 leader_election.cc:302] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.726630 22038 raft_consensus.cc:482] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.726753 22038 raft_consensus.cc:504] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.727066 21791 raft_consensus.cc:2726] T 1c251cfdbec94259be5afbb1b7dbaa38 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.727273 22038 leader_election.cc:288] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
W20240502 14:15:26.728843 15957 leader_election.cc:334] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.730408 13044 catalog_manager.cc:5484] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 1 to 2, leader changed from 8b47d4390f8343a493d6c00123785968 (127.6.196.200) to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 2 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:26.731081 13578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac1154570d4420ae8764ca18667f29" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "7eb4ecb2379b44c0954f2658c4a04350" is_pre_election: true
I20240502 14:15:26.731063 13043 catalog_manager.cc:5484] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 1 to 2, leader changed from 8b47d4390f8343a493d6c00123785968 (127.6.196.200) to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 2 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:26.731544 15955 leader_election.cc:302] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 5297e816f782430d9a01220f0471a882, 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:15:26.731508 13043 catalog_manager.cc:5484] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 2 to 3, leader changed from 8b47d4390f8343a493d6c00123785968 (127.6.196.200) to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 3 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:26.731776 22038 raft_consensus.cc:2726] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:26.734483 13121 consensus_peers.cc:462] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.737408 21931 raft_consensus.cc:482] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.737543 21931 raft_consensus.cc:504] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.738193 21931 leader_election.cc:288] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:26.739171 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac1154570d4420ae8764ca18667f29" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:26.739394 16018 raft_consensus.cc:2454] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 1.
I20240502 14:15:26.740036 13516 leader_election.cc:302] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, d1da7f11f0604f78934354a992bb0f55; no voters:
W20240502 14:15:26.740244 13516 leader_election.cc:334] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.740504 21931 raft_consensus.cc:2781] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.740577 21931 raft_consensus.cc:482] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.740648 21931 raft_consensus.cc:3037] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.743192 21931 raft_consensus.cc:504] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:26.746270 21587 raft_consensus.cc:482] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.746053 13040 catalog_manager.cc:5484] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 1 to 2, leader changed from c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 2 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: NON_VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:26.746474 21587 raft_consensus.cc:504] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
W20240502 14:15:26.747529 15956 consensus_peers.cc:462] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:26.748050 13122 leader_election.cc:334] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.748593 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "caac1154570d4420ae8764ca18667f29" candidate_uuid: "7eb4ecb2379b44c0954f2658c4a04350" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:26.748771 16018 raft_consensus.cc:3037] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:26.749755 13121 leader_election.cc:334] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.749918 13121 leader_election.cc:302] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.750190 21791 raft_consensus.cc:2726] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.750363 21587 leader_election.cc:288] T 1d8443bed0da49d0b24511a8468c2373 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
I20240502 14:15:26.751060 16018 raft_consensus.cc:2454] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eb4ecb2379b44c0954f2658c4a04350 in term 2.
I20240502 14:15:26.753165 13516 leader_election.cc:302] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:26.753566 21255 raft_consensus.cc:2781] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.753674 21255 raft_consensus.cc:686] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 LEADER]: Becoming Leader. State: Replica: 7eb4ecb2379b44c0954f2658c4a04350, State: Running, Role: LEADER
I20240502 14:15:26.753901 21255 consensus_queue.cc:237] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
W20240502 14:15:26.755232 13516 leader_election.cc:334] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.755411 21022 raft_consensus.cc:482] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.755534 21022 raft_consensus.cc:504] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.743691 21931 leader_election.cc:288] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:26.756376 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb770f077c84cb99985ef59253f133b" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:26.757386 21022 leader_election.cc:288] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
W20240502 14:15:26.757925 13253 leader_election.cc:334] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.758008 13253 leader_election.cc:302] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d; no voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:26.760651 21022 raft_consensus.cc:2726] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.762514 13040 catalog_manager.cc:5484] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 reported cstate change: term changed from 1 to 2, leader changed from 5297e816f782430d9a01220f0471a882 (127.6.196.199) to 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196). New cstate: current_term: 2 leader_uuid: "7eb4ecb2379b44c0954f2658c4a04350" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:26.773551 21587 raft_consensus.cc:482] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.773675 21587 raft_consensus.cc:504] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:26.774169 21587 leader_election.cc:288] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.774477 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e393a7736b14d5483973f9cb89058f8" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:26.774685 13318 raft_consensus.cc:2454] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 2.
I20240502 14:15:26.775030 13122 leader_election.cc:302] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:26.775301 21587 raft_consensus.cc:2781] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:15:26.775414 21587 raft_consensus.cc:482] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Starting leader election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:26.775549 21587 raft_consensus.cc:3037] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:15:26.775718 13121 leader_election.cc:334] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.778151 21587 raft_consensus.cc:504] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:26.778714 21587 leader_election.cc:288] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 3 election: Requested vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.778915 13318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e393a7736b14d5483973f9cb89058f8" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 3 candidate_status { last_received { term: 2 index: 10 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:26.779043 13318 raft_consensus.cc:3037] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:26.780812 13318 raft_consensus.cc:2454] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a967bad5504b1e9c6fb720ecdd1625 in term 3.
I20240502 14:15:26.781672 13122 leader_election.cc:302] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:26.782323 21587 raft_consensus.cc:2781] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:26.782474 21587 raft_consensus.cc:686] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 3 LEADER]: Becoming Leader. State: Replica: 20a967bad5504b1e9c6fb720ecdd1625, State: Running, Role: LEADER
W20240502 14:15:26.782539 13121 leader_election.cc:334] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.782775 21587 consensus_queue.cc:237] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:26.786612 13040 catalog_manager.cc:5484] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 reported cstate change: term changed from 2 to 3, leader changed from c86f7440570f4796babfd44c3d7fd847 (127.6.196.198) to 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "20a967bad5504b1e9c6fb720ecdd1625" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240502 14:15:26.805531 20475 raft_consensus.cc:482] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.805650 20475 raft_consensus.cc:504] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:26.806133 20475 leader_election.cc:288] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
W20240502 14:15:26.807322 13384 leader_election.cc:334] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:26.807660 13383 leader_election.cc:334] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.807755 13383 leader_election.cc:302] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.807952 20475 raft_consensus.cc:2726] T f9e6bf2985714a02a6e59d4c48c0a253 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.826411 21255 raft_consensus.cc:482] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.826576 21255 raft_consensus.cc:504] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:26.827097 21255 leader_election.cc:288] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
W20240502 14:15:26.828445 13516 leader_election.cc:334] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:26.828645 13515 leader_election.cc:334] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.828729 13515 leader_election.cc:302] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:26.828974 21255 raft_consensus.cc:2726] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.837426 21791 raft_consensus.cc:482] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.837596 21791 raft_consensus.cc:504] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:26.838203 21791 leader_election.cc:288] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:26.838773 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c871ce05c0804c2fbb3cb06049e6d56a" candidate_uuid: "20a967bad5504b1e9c6fb720ecdd1625" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
W20240502 14:15:26.839387 13122 leader_election.cc:334] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.839505 13122 leader_election.cc:302] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: 5297e816f782430d9a01220f0471a882, d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:26.839784 21791 raft_consensus.cc:2726] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:26.854214 15956 consensus_peers.cc:485] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.860921 13185 raft_consensus.cc:1261] T ba8299c5f722440e92803100e7240723 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20240502 14:15:26.861416 21566 consensus_queue.cc:1035] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:15:26.865535 13252 consensus_peers.cc:485] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.867468 21869 raft_consensus.cc:482] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.867630 21869 raft_consensus.cc:504] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.868088 21869 leader_election.cc:288] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
W20240502 14:15:26.869274 15957 leader_election.cc:334] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.873328 20475 raft_consensus.cc:482] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.873471 20475 raft_consensus.cc:504] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:26.874110 20475 leader_election.cc:288] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:26.875020 13319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a42ded434514cf28e578eb0046dc5f5" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
W20240502 14:15:26.876080 13384 leader_election.cc:334] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.876183 13384 leader_election.cc:302] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: 5297e816f782430d9a01220f0471a882, cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:26.876387 20475 raft_consensus.cc:2726] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.876566 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb770f077c84cb99985ef59253f133b" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d" is_pre_election: true
I20240502 14:15:26.876729 13315 raft_consensus.cc:2454] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:26.880123 15957 leader_election.cc:302] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d, d1da7f11f0604f78934354a992bb0f55; no voters: 5297e816f782430d9a01220f0471a882
I20240502 14:15:26.880411 21869 raft_consensus.cc:2781] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.880503 21869 raft_consensus.cc:482] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.880611 21869 raft_consensus.cc:3037] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.883373 21869 raft_consensus.cc:504] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.883927 21869 leader_election.cc:288] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193)
W20240502 14:15:26.885979 15957 leader_election.cc:334] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.886472 13315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2cb770f077c84cb99985ef59253f133b" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "cd67493f6f0142f896010f2b8a39887d"
I20240502 14:15:26.886628 13315 raft_consensus.cc:3037] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.888439 13315 raft_consensus.cc:2454] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 2.
I20240502 14:15:26.888839 15957 leader_election.cc:302] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cd67493f6f0142f896010f2b8a39887d, d1da7f11f0604f78934354a992bb0f55; no voters: 5297e816f782430d9a01220f0471a882
I20240502 14:15:26.889261 21869 raft_consensus.cc:2781] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.889449 21869 raft_consensus.cc:686] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [term 2 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:26.889837 21869 consensus_queue.cc:237] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } }
I20240502 14:15:26.898595 13040 catalog_manager.cc:5484] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 1 to 2, leader changed from 5297e816f782430d9a01220f0471a882 (127.6.196.199) to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 2 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:26.903456 21255 raft_consensus.cc:482] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.904654 21869 raft_consensus.cc:482] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.904822 21869 raft_consensus.cc:504] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:26.905833 21869 leader_election.cc:288] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:26.906647 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c871ce05c0804c2fbb3cb06049e6d56a" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625" is_pre_election: true
I20240502 14:15:26.907138 13185 raft_consensus.cc:2454] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 1.
I20240502 14:15:26.907588 15955 leader_election.cc:302] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters:
I20240502 14:15:26.908035 21869 raft_consensus.cc:2781] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.908169 21869 raft_consensus.cc:482] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.908295 21869 raft_consensus.cc:3037] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.911309 21869 raft_consensus.cc:504] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
W20240502 14:15:26.912271 15957 leader_election.cc:334] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.912259 13185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c871ce05c0804c2fbb3cb06049e6d56a" candidate_uuid: "d1da7f11f0604f78934354a992bb0f55" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "20a967bad5504b1e9c6fb720ecdd1625"
I20240502 14:15:26.912443 13185 raft_consensus.cc:3037] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:26.914093 15957 leader_election.cc:334] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.915107 13185 raft_consensus.cc:2454] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d1da7f11f0604f78934354a992bb0f55 in term 2.
I20240502 14:15:26.915467 15955 leader_election.cc:302] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625, d1da7f11f0604f78934354a992bb0f55; no voters: 5297e816f782430d9a01220f0471a882
I20240502 14:15:26.915709 22038 raft_consensus.cc:2781] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.915740 21869 leader_election.cc:288] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 election: Requested vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687)
I20240502 14:15:26.915846 22038 raft_consensus.cc:686] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [term 2 LEADER]: Becoming Leader. State: Replica: d1da7f11f0604f78934354a992bb0f55, State: Running, Role: LEADER
I20240502 14:15:26.916101 22038 consensus_queue.cc:237] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } }
I20240502 14:15:26.920167 21022 raft_consensus.cc:482] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.920313 21022 raft_consensus.cc:504] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:26.920864 21022 leader_election.cc:288] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:26.921438 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a42ded434514cf28e578eb0046dc5f5" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3" is_pre_election: true
I20240502 14:15:26.921602 13445 raft_consensus.cc:2454] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 1.
I20240502 14:15:26.922565 13251 leader_election.cc:302] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters:
I20240502 14:15:26.923023 21022 raft_consensus.cc:2781] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240502 14:15:26.923087 21022 raft_consensus.cc:482] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.923141 21022 raft_consensus.cc:3037] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:26.923540 13253 leader_election.cc:334] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.925412 21022 raft_consensus.cc:504] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:26.925901 21022 leader_election.cc:288] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: Requested vote from peers 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
W20240502 14:15:26.927100 13253 leader_election.cc:334] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.928309 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9a42ded434514cf28e578eb0046dc5f5" candidate_uuid: "cd67493f6f0142f896010f2b8a39887d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "88b77f6edc1c4a72b548bf47c05704f3"
I20240502 14:15:26.928455 13445 raft_consensus.cc:3037] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:26.931583 13445 raft_consensus.cc:2454] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cd67493f6f0142f896010f2b8a39887d in term 2.
I20240502 14:15:26.932032 13251 leader_election.cc:302] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, cd67493f6f0142f896010f2b8a39887d; no voters: 5297e816f782430d9a01220f0471a882
I20240502 14:15:26.932565 21932 raft_consensus.cc:2781] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:26.932673 21932 raft_consensus.cc:686] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [term 2 LEADER]: Becoming Leader. State: Replica: cd67493f6f0142f896010f2b8a39887d, State: Running, Role: LEADER
I20240502 14:15:26.932904 21932 consensus_queue.cc:237] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:26.942448 13040 catalog_manager.cc:5484] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d reported cstate change: term changed from 1 to 2, leader changed from 5297e816f782430d9a01220f0471a882 (127.6.196.199) to cd67493f6f0142f896010f2b8a39887d (127.6.196.194). New cstate: current_term: 2 leader_uuid: "cd67493f6f0142f896010f2b8a39887d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:26.947163 21255 raft_consensus.cc:504] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
W20240502 14:15:26.949146 13516 leader_election.cc:334] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:26.949486 13515 leader_election.cc:334] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.949561 13515 leader_election.cc:302] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.949774 13581 raft_consensus.cc:1261] T f961cf70d8ca482ba331241052833fd3 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:26.949792 21931 raft_consensus.cc:2726] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.950090 21255 leader_election.cc:288] T 5348ba5435c74f509ed77993191ef002 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
I20240502 14:15:26.950556 21022 consensus_queue.cc:1035] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:26.956872 13043 catalog_manager.cc:5484] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 reported cstate change: term changed from 1 to 2, leader changed from 5297e816f782430d9a01220f0471a882 (127.6.196.199) to d1da7f11f0604f78934354a992bb0f55 (127.6.196.197). New cstate: current_term: 2 leader_uuid: "d1da7f11f0604f78934354a992bb0f55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:26.958967 13252 consensus_peers.cc:485] T f961cf70d8ca482ba331241052833fd3 P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:26.966527 21791 raft_consensus.cc:482] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:26.966678 21791 raft_consensus.cc:504] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: false } }
I20240502 14:15:26.967267 21791 leader_election.cc:288] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
W20240502 14:15:26.968640 13122 leader_election.cc:334] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:26.969081 13121 leader_election.cc:334] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:26.969281 13121 leader_election.cc:302] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 20a967bad5504b1e9c6fb720ecdd1625; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:26.969635 21791 raft_consensus.cc:2726] T e95022a368ec47949d1003c39a7fe433 P 20a967bad5504b1e9c6fb720ecdd1625 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:26.972046 13581 raft_consensus.cc:1261] T 7a740ff5e0964efcb1fd7dd54dc4094c P 7eb4ecb2379b44c0954f2658c4a04350 [term 3 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:26.972522 22038 consensus_queue.cc:1035] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:27.011622 20475 raft_consensus.cc:482] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:27.011795 20475 raft_consensus.cc:504] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { promote: false } }
I20240502 14:15:27.012377 20475 leader_election.cc:288] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
W20240502 14:15:27.013363 13384 leader_election.cc:334] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:27.013729 13383 leader_election.cc:334] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:27.013823 13383 leader_election.cc:302] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:27.014039 21876 raft_consensus.cc:2726] T c78403038b614e6db2afb617328f6411 P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:27.024075 13515 consensus_peers.cc:485] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.033452 13581 raft_consensus.cc:1261] T c1b7405d7ff34356b0aa6ecb12b50817 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:15:27.034366 21587 consensus_queue.cc:1035] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:27.039597 13185 raft_consensus.cc:1261] T 9535cdbc4294429593012b088528c449 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240502 14:15:27.040409 13121 consensus_peers.cc:485] T c1b7405d7ff34356b0aa6ecb12b50817 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.041193 21487 consensus_queue.cc:1035] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240502 14:15:27.044459 13383 consensus_peers.cc:485] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.047444 13516 consensus_peers.cc:462] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:27.053913 15956 consensus_peers.cc:485] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.103853 13447 raft_consensus.cc:1261] T df55a71945d34af0affa6eccbdf84dae P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240502 14:15:27.104400 21487 consensus_queue.cc:1035] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240502 14:15:27.115350 13315 raft_consensus.cc:1261] T caa4af5f41674aaa8587158f25e495fb P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:27.115885 21876 consensus_queue.cc:1035] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:15:27.118273 15956 consensus_peers.cc:485] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.119025 15956 consensus_peers.cc:485] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.121610 13121 consensus_peers.cc:485] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.129395 16018 raft_consensus.cc:1261] T c3e9160c46d64fa0a964cc39aec91dfd P d1da7f11f0604f78934354a992bb0f55 [term 3 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 14:15:27.132364 13185 raft_consensus.cc:1261] T 8a3b25a9e8b049a88bb999d9362e98da P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer 88b77f6edc1c4a72b548bf47c05704f3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:15:27.132910 21876 consensus_queue.cc:1035] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:27.133026 21707 consensus_queue.cc:1035] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240502 14:15:27.134660 13252 consensus_peers.cc:485] T c3e9160c46d64fa0a964cc39aec91dfd P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.136967 13121 consensus_peers.cc:485] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.151533 13384 consensus_peers.cc:485] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.156875 15956 consensus_peers.cc:462] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:27.158058 13318 raft_consensus.cc:1261] T 990da252c05c45d8b7a3e47288f1ab61 P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:27.158596 21791 consensus_queue.cc:1035] T 990da252c05c45d8b7a3e47288f1ab61 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:15:27.160784 15956 consensus_peers.cc:485] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.175525 13581 raft_consensus.cc:1261] T 6931796f283a4f20b7fec8c3163b8444 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:15:27.176383 22033 consensus_queue.cc:1035] T 6931796f283a4f20b7fec8c3163b8444 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:27.176789 16018 raft_consensus.cc:1261] T caac1154570d4420ae8764ca18667f29 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:27.177350 21931 consensus_queue.cc:1035] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:27.191880 13186 raft_consensus.cc:1261] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:27.192584 21487 consensus_queue.cc:1035] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/rebalancer_tool-test.cc:1203: Failure
Value of: s.ok()
Actual: false
Expected: true
Runtime error: /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 45
Average Replica Count | 22.500000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 2
Maximum | 3
Average | 2.950000
stderr: W20240502 14:14:50.195847 16474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:14:50.196211 16474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:14:50.204718 16474 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
E20240502 14:14:50.614575 16551 ksck_remote.cc:634] unable to open table rebalancer_dynamic_table: Not found: the table does not exist: table_name: "rebalancer_dynamic_table"
I20240502 14:14:51.253784 16474 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240502 14:14:51.446763 16684 meta_cache.cc:788] Not found: LookupRpcById { tablet: '9b2f3557f96d4f33b02941d086de1391', attempt: 1 } failed
W20240502 14:14:52.032672 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 160fdc60b8f84393830bdcb0e0d4769d: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.032963 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b7310fe1c01b4fe7b29328321c07e72d: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.033115 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cbc2cd67f3c24195825b542cf2878195: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:52.033186 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4f0d325b9ba44b7c98e5cc0c7452223a: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.033274 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 160fdc60b8f84393830bdcb0e0d4769d: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:52.033355 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 90f309e38a4949de93b82598a1e7e310: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.033509 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 75d4a7920a3643c5adb07f84cba064bd: could not find any suitable replica to move from server 20a967bad5504b1e9c6fb720ecdd1625 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:52.033608 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3da4fce4919b44df830cd0c2062a894e: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:52.033735 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe675f309a11456aaf1fc329cfdb1707: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.033875 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6607ab63fd0e452e8f7c02918e77ea9e: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.033983 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cbc2cd67f3c24195825b542cf2878195: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.034089 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4f0d325b9ba44b7c98e5cc0c7452223a: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:52.034242 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0cb4710334148e4b976992ee52cd47c: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:52.034346 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eed0aa41c10e41e79b79d17916bc3464: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.034409 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c97a45b4ee8c48a38ebd0709da5770f7: could not find any suitable replica to move from server 20a967bad5504b1e9c6fb720ecdd1625 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:52.034471 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9fb2dba02f0745699edbcaa1c4b1f4ed: could not find any suitable replica to move from server 20a967bad5504b1e9c6fb720ecdd1625 to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:52.034554 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 557f9d603bda4eb385d26221f3612c33: could not find any suitable replica to move from server 20a967bad5504b1e9c6fb720ecdd1625 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:52.034682 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe3346248b4546db98fb1fc546016280: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:52.034842 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ce45faa68624e3bb49f45d1ad39a0e5: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:52.034999 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8fa288258a0f4231be6de034088616ab: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:52.035108 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6607ab63fd0e452e8f7c02918e77ea9e: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:52.035192 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 160fdc60b8f84393830bdcb0e0d4769d: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.035248 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c97a45b4ee8c48a38ebd0709da5770f7: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:52.035308 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 90f309e38a4949de93b82598a1e7e310: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:52.035362 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0cb4710334148e4b976992ee52cd47c: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server d1da7f11f0604f78934354a992bb0f55
I20240502 14:14:52.039634 16474 rebalancer_tool.cc:1084] tablet 8e96c49666584fceb759383d24ac1c4a: 'cd67493f6f0142f896010f2b8a39887d' -> '8b47d4390f8343a493d6c00123785968' move ignored: Not found: 8e96c49666584fceb759383d24ac1c4a: tablet not found
I20240502 14:14:52.039762 16474 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:14:52.856755 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 75d4a7920a3643c5adb07f84cba064bd: could not find any suitable replica to move from server 20a967bad5504b1e9c6fb720ecdd1625 to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:52.857798 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 75d4a7920a3643c5adb07f84cba064bd: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.858125 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0cb4710334148e4b976992ee52cd47c: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:52.858211 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9fb2dba02f0745699edbcaa1c4b1f4ed: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.858415 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 90f309e38a4949de93b82598a1e7e310: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:52.858582 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table eed0aa41c10e41e79b79d17916bc3464: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.858824 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 4f0d325b9ba44b7c98e5cc0c7452223a: could not find any suitable replica to move from server 20a967bad5504b1e9c6fb720ecdd1625 to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:52.859009 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c97a45b4ee8c48a38ebd0709da5770f7: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:52.859210 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3da4fce4919b44df830cd0c2062a894e: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:52.859270 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 75d4a7920a3643c5adb07f84cba064bd: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:52.859449 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7a3f9a47775b46e38274ca5c7b2dbe6d: could not find any suitable replica to move from server 8b47d4390f8343a493d6c00123785968 to server cd67493f6f0142f896010f2b8a39887d
W20240502 14:14:52.859587 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ce45faa68624e3bb49f45d1ad39a0e5: could not find any suitable replica to move from server 20a967bad5504b1e9c6fb720ecdd1625 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:52.859871 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cbc2cd67f3c24195825b542cf2878195: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:52.860005 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 160fdc60b8f84393830bdcb0e0d4769d: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:52.860128 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe3346248b4546db98fb1fc546016280: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.860216 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 557f9d603bda4eb385d26221f3612c33: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:52.860318 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 75d4a7920a3643c5adb07f84cba064bd: could not find any suitable replica to move from server 20a967bad5504b1e9c6fb720ecdd1625 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:52.860412 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8fa288258a0f4231be6de034088616ab: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:52.860494 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe675f309a11456aaf1fc329cfdb1707: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:52.860574 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b7310fe1c01b4fe7b29328321c07e72d: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:52.860695 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 53e326f05b5545f58887e5f17abc4ab7: could not find any suitable replica to move from server 20a967bad5504b1e9c6fb720ecdd1625 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:52.860778 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 90f309e38a4949de93b82598a1e7e310: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:52.861049 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0cb4710334148e4b976992ee52cd47c: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:52.861161 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9fb2dba02f0745699edbcaa1c4b1f4ed: could not find any suitable replica to move from server 20a967bad5504b1e9c6fb720ecdd1625 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:52.861249 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cbc2cd67f3c24195825b542cf2878195: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:52.861339 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 160fdc60b8f84393830bdcb0e0d4769d: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server c86f7440570f4796babfd44c3d7fd847
I20240502 14:14:52.911967 16474 rebalancer_tool.cc:1074] tablet dcf9e4934b4d4400884eee33f42b29f5: '20a967bad5504b1e9c6fb720ecdd1625' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:14:52.995007 16474 rebalancer_tool.cc:1074] tablet ba519bb3fb534e7fb6106f402e27446d: '88b77f6edc1c4a72b548bf47c05704f3' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:14:53.038776 16474 rebalancer_tool.cc:1074] tablet 718223dd8fd3432cb174a53325dc784c: '7eb4ecb2379b44c0954f2658c4a04350' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:14:53.075385 16474 rebalancer_tool.cc:1074] tablet bbd12f1c9884497b82c941b7231645bb: 'cd67493f6f0142f896010f2b8a39887d' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:14:53.186185 16474 rebalancer_tool.cc:1074] tablet 3ab40fe08f714c99a5961cc2ad0915d3: '20a967bad5504b1e9c6fb720ecdd1625' -> '88b77f6edc1c4a72b548bf47c05704f3' move scheduled
I20240502 14:14:53.213732 16474 rebalancer_tool.cc:1074] tablet eab73d39b1f7455bba7aef5e515805c7: 'd1da7f11f0604f78934354a992bb0f55' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:14:53.247001 16474 rebalancer_tool.cc:1074] tablet f966f36415994f9eaa8046b407135761: '7eb4ecb2379b44c0954f2658c4a04350' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:14:53.356094 16474 rebalancer_tool.cc:1074] tablet 0ec7ae12c17247e9b8c03659435bd4e2: 'cd67493f6f0142f896010f2b8a39887d' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:14:53.625169 16474 rebalancer_tool.cc:1074] tablet e3be42ac139741d19dd872ab551bebae: '20a967bad5504b1e9c6fb720ecdd1625' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:14:53.873558 16474 rebalancer_tool.cc:1074] tablet d5975e02205a43d5b93ed8ab51f8a0f8: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:14:53.876396 16474 rebalancer_tool.cc:1084] tablet 373143a499c0492991c0693caf245ff4: '7eb4ecb2379b44c0954f2658c4a04350' -> '8b47d4390f8343a493d6c00123785968' move ignored: Not found: 373143a499c0492991c0693caf245ff4: tablet not found
I20240502 14:14:53.876494 16474 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:14:55.301355 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ce45faa68624e3bb49f45d1ad39a0e5: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:55.301582 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9fb2dba02f0745699edbcaa1c4b1f4ed: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:55.301826 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c97a45b4ee8c48a38ebd0709da5770f7: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:55.301952 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cbc2cd67f3c24195825b542cf2878195: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:55.302027 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 160fdc60b8f84393830bdcb0e0d4769d: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:55.302137 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ce45faa68624e3bb49f45d1ad39a0e5: could not find any suitable replica to move from server 20a967bad5504b1e9c6fb720ecdd1625 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:55.302211 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 557f9d603bda4eb385d26221f3612c33: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:55.302294 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0cb4710334148e4b976992ee52cd47c: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:55.302453 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9fb2dba02f0745699edbcaa1c4b1f4ed: could not find any suitable replica to move from server 20a967bad5504b1e9c6fb720ecdd1625 to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:55.302604 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table cb136ab1e407401daef8e9856e8ee6f9: could not find any suitable replica to move from server 5297e816f782430d9a01220f0471a882 to server 20a967bad5504b1e9c6fb720ecdd1625
W20240502 14:14:55.302776 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table b7310fe1c01b4fe7b29328321c07e72d: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:55.302937 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 90f309e38a4949de93b82598a1e7e310: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:55.303012 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 1ce45faa68624e3bb49f45d1ad39a0e5: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:55.303114 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe675f309a11456aaf1fc329cfdb1707: could not find any suitable replica to move from server 20a967bad5504b1e9c6fb720ecdd1625 to server 5297e816f782430d9a01220f0471a882
W20240502 14:14:55.303208 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8fa288258a0f4231be6de034088616ab: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server d1da7f11f0604f78934354a992bb0f55
W20240502 14:14:55.303288 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3da4fce4919b44df830cd0c2062a894e: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:55.303403 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table a6584ce357f343ffa6c92e7412b11020: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server 8b47d4390f8343a493d6c00123785968
W20240502 14:14:55.303500 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 90f309e38a4949de93b82598a1e7e310: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:14:55.303617 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c0cb4710334148e4b976992ee52cd47c: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server 5297e816f782430d9a01220f0471a882
I20240502 14:14:55.348416 16474 rebalancer_tool.cc:1074] tablet ba8299c5f722440e92803100e7240723: '7eb4ecb2379b44c0954f2658c4a04350' -> 'cd67493f6f0142f896010f2b8a39887d' move scheduled
I20240502 14:14:55.407918 16474 rebalancer_tool.cc:1074] tablet 16d6faae2add42c5bd4af6018a13b841: '5297e816f782430d9a01220f0471a882' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:14:55.468961 16474 rebalancer_tool.cc:1074] tablet 8a6b71f7e77b4084a8904b1045099472: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:14:55.516412 16474 rebalancer_tool.cc:1074] tablet c3e9160c46d64fa0a964cc39aec91dfd: '20a967bad5504b1e9c6fb720ecdd1625' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:14:55.541458 16474 rebalancer_tool.cc:1074] tablet 9a95b9049a344b98be91b9961f3e5bee: '7eb4ecb2379b44c0954f2658c4a04350' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:14:55.610854 16474 rebalancer_tool.cc:1074] tablet 7852539de07744caaf385620c5166861: 'cd67493f6f0142f896010f2b8a39887d' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:14:55.701690 16474 rebalancer_tool.cc:1074] tablet b2905b7940fb45c49ef8b18943dae256: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:14:55.978864 16474 rebalancer_tool.cc:1074] tablet abd272455c064ba48726d933d53c83ed: '20a967bad5504b1e9c6fb720ecdd1625' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:14:56.076869 16474 rebalancer_tool.cc:1074] tablet f9c268e5a8e7462ab8e940775c73e591: '7eb4ecb2379b44c0954f2658c4a04350' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:14:56.382889 16474 rebalancer_tool.cc:1074] tablet 652e49046212458280095c016a342f47: 'cd67493f6f0142f896010f2b8a39887d' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:14:56.465742 16474 rebalancer_tool.cc:1123] tablet f966f36415994f9eaa8046b407135761: '7eb4ecb2379b44c0954f2658c4a04350' -> 'c86f7440570f4796babfd44c3d7fd847' move is abandoned: Not found: f966f36415994f9eaa8046b407135761: tablet not found
I20240502 14:14:56.466750 16474 rebalancer_tool.cc:1123] tablet dcf9e4934b4d4400884eee33f42b29f5: '20a967bad5504b1e9c6fb720ecdd1625' -> 'd1da7f11f0604f78934354a992bb0f55' move is abandoned: Not found: dcf9e4934b4d4400884eee33f42b29f5: tablet not found
I20240502 14:14:56.494622 16474 rebalancer_tool.cc:1136] tablet e3be42ac139741d19dd872ab551bebae: '20a967bad5504b1e9c6fb720ecdd1625' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:14:56.510237 16474 rebalancer_tool.cc:1136] tablet eab73d39b1f7455bba7aef5e515805c7: 'd1da7f11f0604f78934354a992bb0f55' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:14:56.771911 16474 rebalancer_tool.cc:1136] tablet 3ab40fe08f714c99a5961cc2ad0915d3: '20a967bad5504b1e9c6fb720ecdd1625' -> '88b77f6edc1c4a72b548bf47c05704f3' move completed: OK
I20240502 14:14:57.342218 16474 rebalancer_tool.cc:1136] tablet ba519bb3fb534e7fb6106f402e27446d: '88b77f6edc1c4a72b548bf47c05704f3' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:14:57.608870 16474 rebalancer_tool.cc:1136] tablet 0ec7ae12c17247e9b8c03659435bd4e2: 'cd67493f6f0142f896010f2b8a39887d' -> '5297e816f782430d9a01220f0471a882' move completed: OK
I20240502 14:14:57.691642 16474 rebalancer_tool.cc:1136] tablet ba8299c5f722440e92803100e7240723: '7eb4ecb2379b44c0954f2658c4a04350' -> 'cd67493f6f0142f896010f2b8a39887d' move completed: OK
I20240502 14:14:57.833375 16474 rebalancer_tool.cc:1136] tablet f9c268e5a8e7462ab8e940775c73e591: '7eb4ecb2379b44c0954f2658c4a04350' -> '5297e816f782430d9a01220f0471a882' move completed: OK
I20240502 14:14:57.834476 16474 rebalancer_tool.cc:1123] tablet 718223dd8fd3432cb174a53325dc784c: '7eb4ecb2379b44c0954f2658c4a04350' -> 'c86f7440570f4796babfd44c3d7fd847' move is abandoned: Not found: 718223dd8fd3432cb174a53325dc784c: tablet not found
I20240502 14:14:57.882910 16474 rebalancer_tool.cc:1136] tablet d5975e02205a43d5b93ed8ab51f8a0f8: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move completed: OK
I20240502 14:14:57.973337 16474 rebalancer_tool.cc:1136] tablet 16d6faae2add42c5bd4af6018a13b841: '5297e816f782430d9a01220f0471a882' -> 'c86f7440570f4796babfd44c3d7fd847' move completed: OK
I20240502 14:14:57.976050 16474 rebalancer_tool.cc:1123] tablet bbd12f1c9884497b82c941b7231645bb: 'cd67493f6f0142f896010f2b8a39887d' -> '5297e816f782430d9a01220f0471a882' move is abandoned: Not found: bbd12f1c9884497b82c941b7231645bb: tablet not found
I20240502 14:14:57.985018 16474 rebalancer_tool.cc:1136] tablet 9a95b9049a344b98be91b9961f3e5bee: '7eb4ecb2379b44c0954f2658c4a04350' -> '5297e816f782430d9a01220f0471a882' move completed: OK
I20240502 14:14:57.992533 16474 rebalancer_tool.cc:1136] tablet c3e9160c46d64fa0a964cc39aec91dfd: '20a967bad5504b1e9c6fb720ecdd1625' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:14:58.000142 16474 rebalancer_tool.cc:1136] tablet 7852539de07744caaf385620c5166861: 'cd67493f6f0142f896010f2b8a39887d' -> 'c86f7440570f4796babfd44c3d7fd847' move completed: OK
I20240502 14:14:58.026093 16474 rebalancer_tool.cc:1074] tablet 0854e9f4851e400f9af18181a71ae8f7: '88b77f6edc1c4a72b548bf47c05704f3' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:14:58.053740 16474 rebalancer_tool.cc:1074] tablet 27857198d4ec430ebee897c90141c58d: 'cd67493f6f0142f896010f2b8a39887d' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:14:58.093678 16474 rebalancer_tool.cc:1074] tablet ab51b049c3674b629b55ed29df5b1fab: '7eb4ecb2379b44c0954f2658c4a04350' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:14:58.129783 16474 rebalancer_tool.cc:1074] tablet db06293ca0914bb88dbff3808e3f1874: '20a967bad5504b1e9c6fb720ecdd1625' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:14:58.182472 16474 rebalancer_tool.cc:1074] tablet f5bf499e8c044a1886989a90c6bb69fa: '8b47d4390f8343a493d6c00123785968' -> '88b77f6edc1c4a72b548bf47c05704f3' move scheduled
I20240502 14:14:58.356029 16474 rebalancer_tool.cc:1074] tablet f961cf70d8ca482ba331241052833fd3: 'c86f7440570f4796babfd44c3d7fd847' -> 'cd67493f6f0142f896010f2b8a39887d' move scheduled
I20240502 14:14:58.491967 16474 rebalancer_tool.cc:1074] tablet 536c22c444344bb485d3e522f0f77154: '5297e816f782430d9a01220f0471a882' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:14:58.713573 16474 rebalancer_tool.cc:1074] tablet 1e2b8c72c7ba4be58f31ac073e41f2a8: '8b47d4390f8343a493d6c00123785968' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:14:59.119185 16474 rebalancer_tool.cc:1074] tablet 766f50e1e5a44c8ba211a446bcf86705: '88b77f6edc1c4a72b548bf47c05704f3' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:14:59.255342 16474 rebalancer_tool.cc:1074] tablet 80040c60b029481285c50171bcc5ce35: 'cd67493f6f0142f896010f2b8a39887d' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:14:59.510327 16474 rebalancer_tool.cc:1136] tablet 652e49046212458280095c016a342f47: 'cd67493f6f0142f896010f2b8a39887d' -> 'c86f7440570f4796babfd44c3d7fd847' move completed: OK
I20240502 14:14:59.523375 16474 rebalancer_tool.cc:1136] tablet abd272455c064ba48726d933d53c83ed: '20a967bad5504b1e9c6fb720ecdd1625' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:14:59.534132 16474 rebalancer_tool.cc:1136] tablet b2905b7940fb45c49ef8b18943dae256: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move completed: OK
I20240502 14:14:59.543336 16474 rebalancer_tool.cc:1136] tablet 8a6b71f7e77b4084a8904b1045099472: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move completed: OK
I20240502 14:14:59.566864 16474 rebalancer_tool.cc:1074] tablet 9ad2eb265e0f469787eb1699e339e0ed: '20a967bad5504b1e9c6fb720ecdd1625' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:14:59.678028 16474 rebalancer_tool.cc:1074] tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a: '88b77f6edc1c4a72b548bf47c05704f3' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:14:59.684178 16474 rebalancer_tool.cc:1084] tablet b3b57323bd14448b9406ce53ada769e5: 'cd67493f6f0142f896010f2b8a39887d' -> 'd1da7f11f0604f78934354a992bb0f55' move ignored: Not found: b3b57323bd14448b9406ce53ada769e5: tablet not found
I20240502 14:14:59.684326 16474 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240502 14:15:01.043368 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9dc6b07ff15645ab9222cc55c5a4a00b: could not find any suitable replica to move from server cd67493f6f0142f896010f2b8a39887d to server 20a967bad5504b1e9c6fb720ecdd1625
W20240502 14:15:01.043709 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3da4fce4919b44df830cd0c2062a894e: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:15:01.044354 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 71087d0bec7e450da33bd69a7f555355: could not find any suitable replica to move from server c86f7440570f4796babfd44c3d7fd847 to server 7eb4ecb2379b44c0954f2658c4a04350
W20240502 14:15:01.045388 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9dc6b07ff15645ab9222cc55c5a4a00b: could not find any suitable replica to move from server 5297e816f782430d9a01220f0471a882 to server 88b77f6edc1c4a72b548bf47c05704f3
W20240502 14:15:01.045773 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table c97a45b4ee8c48a38ebd0709da5770f7: could not find any suitable replica to move from server 88b77f6edc1c4a72b548bf47c05704f3 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:15:01.046335 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9dc6b07ff15645ab9222cc55c5a4a00b: could not find any suitable replica to move from server c86f7440570f4796babfd44c3d7fd847 to server 7eb4ecb2379b44c0954f2658c4a04350
W20240502 14:15:01.046466 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8fa288258a0f4231be6de034088616ab: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server 5297e816f782430d9a01220f0471a882
W20240502 14:15:01.046592 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fe675f309a11456aaf1fc329cfdb1707: could not find any suitable replica to move from server 7eb4ecb2379b44c0954f2658c4a04350 to server d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:01.409957 16474 rebalancer_tool.cc:1074] tablet 819e6e83e6064c539cfc73a28c43db5b: 'd1da7f11f0604f78934354a992bb0f55' -> '7eb4ecb2379b44c0954f2658c4a04350' move scheduled
I20240502 14:15:01.511477 16474 rebalancer_tool.cc:1074] tablet 7852539de07744caaf385620c5166861: '88b77f6edc1c4a72b548bf47c05704f3' -> '7eb4ecb2379b44c0954f2658c4a04350' move scheduled
I20240502 14:15:01.545944 16474 rebalancer_tool.cc:1074] tablet 1b34f2a22306459784a1ad7b81cd1683: 'c86f7440570f4796babfd44c3d7fd847' -> 'cd67493f6f0142f896010f2b8a39887d' move scheduled
I20240502 14:15:01.576025 16474 rebalancer_tool.cc:1074] tablet 3fa51110016d4cafafbe25a1bd213f2e: '20a967bad5504b1e9c6fb720ecdd1625' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:15:01.614207 16474 rebalancer_tool.cc:1074] tablet 2e51feadc94c48e9bcecffecacbda4b9: '7eb4ecb2379b44c0954f2658c4a04350' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:15:01.683684 16474 rebalancer_tool.cc:1074] tablet b648703b8fc742ecbe1dfd2047d4e68d: 'cd67493f6f0142f896010f2b8a39887d' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:15:01.729838 16474 rebalancer_tool.cc:1074] tablet eb27b3bb29754d7399b0314ada6b1ba7: 'd1da7f11f0604f78934354a992bb0f55' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:15:01.818863 16474 rebalancer_tool.cc:1136] tablet 21ac2a9fa2154a0bbbe15aeaf178ea1a: '88b77f6edc1c4a72b548bf47c05704f3' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:15:01.832372 16474 rebalancer_tool.cc:1136] tablet 9ad2eb265e0f469787eb1699e339e0ed: '20a967bad5504b1e9c6fb720ecdd1625' -> 'c86f7440570f4796babfd44c3d7fd847' move completed: OK
I20240502 14:15:01.851285 16474 rebalancer_tool.cc:1136] tablet 80040c60b029481285c50171bcc5ce35: 'cd67493f6f0142f896010f2b8a39887d' -> '5297e816f782430d9a01220f0471a882' move completed: OK
I20240502 14:15:01.861121 16474 rebalancer_tool.cc:1136] tablet 766f50e1e5a44c8ba211a446bcf86705: '88b77f6edc1c4a72b548bf47c05704f3' -> 'c86f7440570f4796babfd44c3d7fd847' move completed: OK
I20240502 14:15:01.927436 16474 rebalancer_tool.cc:1136] tablet 1e2b8c72c7ba4be58f31ac073e41f2a8: '8b47d4390f8343a493d6c00123785968' -> 'd1da7f11f0604f78934354a992bb0f55' move completed: OK
I20240502 14:15:01.955351 16474 rebalancer_tool.cc:1136] tablet 536c22c444344bb485d3e522f0f77154: '5297e816f782430d9a01220f0471a882' -> '20a967bad5504b1e9c6fb720ecdd1625' move completed: OK
I20240502 14:15:01.964075 16474 rebalancer_tool.cc:1136] tablet f961cf70d8ca482ba331241052833fd3: 'c86f7440570f4796babfd44c3d7fd847' -> 'cd67493f6f0142f896010f2b8a39887d' move completed: OK
I20240502 14:15:01.971868 16474 rebalancer_tool.cc:1136] tablet f5bf499e8c044a1886989a90c6bb69fa: '8b47d4390f8343a493d6c00123785968' -> '88b77f6edc1c4a72b548bf47c05704f3' move completed: OK
I20240502 14:15:01.987366 16474 rebalancer_tool.cc:1136] tablet db06293ca0914bb88dbff3808e3f1874: '20a967bad5504b1e9c6fb720ecdd1625' -> 'd1da7f11f0604f78934354a992bb0f55' move completed: OK
I20240502 14:15:01.994868 16474 rebalancer_tool.cc:1136] tablet ab51b049c3674b629b55ed29df5b1fab: '7eb4ecb2379b44c0954f2658c4a04350' -> '5297e816f782430d9a01220f0471a882' move completed: OK
I20240502 14:15:02.006325 16474 rebalancer_tool.cc:1136] tablet 27857198d4ec430ebee897c90141c58d: 'cd67493f6f0142f896010f2b8a39887d' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:15:02.027559 16474 rebalancer_tool.cc:1136] tablet 0854e9f4851e400f9af18181a71ae8f7: '88b77f6edc1c4a72b548bf47c05704f3' -> '5297e816f782430d9a01220f0471a882' move completed: OK
I20240502 14:15:02.066713 16474 rebalancer_tool.cc:1074] tablet 16c817b864c4425eaa85ac4b5aff59fa: '88b77f6edc1c4a72b548bf47c05704f3' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:15:02.397018 16474 rebalancer_tool.cc:1074] tablet 8e3da1778819497bb3bd56f9ba451046: '8b47d4390f8343a493d6c00123785968' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:15:02.650005 16474 rebalancer_tool.cc:1074] tablet dee785c61c73426fb5665c1334ff372b: 'cd67493f6f0142f896010f2b8a39887d' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:15:02.691690 16474 rebalancer_tool.cc:1074] tablet 0e393a7736b14d5483973f9cb89058f8: '88b77f6edc1c4a72b548bf47c05704f3' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:15:02.718590 16474 rebalancer_tool.cc:1074] tablet ba8299c5f722440e92803100e7240723: 'd1da7f11f0604f78934354a992bb0f55' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:15:02.740913 16474 rebalancer_tool.cc:1074] tablet 8a6b71f7e77b4084a8904b1045099472: '7eb4ecb2379b44c0954f2658c4a04350' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:15:02.806136 16474 rebalancer_tool.cc:1074] tablet 8e69d81920d147a38e2542d22ba8352a: '20a967bad5504b1e9c6fb720ecdd1625' -> 'cd67493f6f0142f896010f2b8a39887d' move scheduled
I20240502 14:15:02.887228 16474 rebalancer_tool.cc:1074] tablet 0a2fc48c889b481e9b4d249819a196a0: '88b77f6edc1c4a72b548bf47c05704f3' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:15:03.554652 16474 rebalancer_tool.cc:1074] tablet 5348ba5435c74f509ed77993191ef002: '8b47d4390f8343a493d6c00123785968' -> '7eb4ecb2379b44c0954f2658c4a04350' move scheduled
I20240502 14:15:03.797685 16474 rebalancer_tool.cc:1074] tablet 0dd07adcbb1f4bd4bcb13bd089c89974: '5297e816f782430d9a01220f0471a882' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:15:04.104569 16474 rebalancer_tool.cc:1074] tablet caa4af5f41674aaa8587158f25e495fb: 'd1da7f11f0604f78934354a992bb0f55' -> 'cd67493f6f0142f896010f2b8a39887d' move scheduled
I20240502 14:15:04.288057 16474 rebalancer_tool.cc:1074] tablet 3d39f694c8554638a3b10ebf7cd01350: '88b77f6edc1c4a72b548bf47c05704f3' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:15:05.484999 16474 rebalancer_tool.cc:1136] tablet 0a2fc48c889b481e9b4d249819a196a0: '88b77f6edc1c4a72b548bf47c05704f3' -> 'c86f7440570f4796babfd44c3d7fd847' move completed: OK
I20240502 14:15:05.507858 16474 rebalancer_tool.cc:1136] tablet 8a6b71f7e77b4084a8904b1045099472: '7eb4ecb2379b44c0954f2658c4a04350' -> '5297e816f782430d9a01220f0471a882' move completed: OK
I20240502 14:15:05.567109 16474 rebalancer_tool.cc:1136] tablet ba8299c5f722440e92803100e7240723: 'd1da7f11f0604f78934354a992bb0f55' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:15:05.739338 16474 rebalancer_tool.cc:1136] tablet 8e3da1778819497bb3bd56f9ba451046: '8b47d4390f8343a493d6c00123785968' -> '20a967bad5504b1e9c6fb720ecdd1625' move completed: OK
I20240502 14:15:05.850726 16474 rebalancer_tool.cc:1136] tablet 16c817b864c4425eaa85ac4b5aff59fa: '88b77f6edc1c4a72b548bf47c05704f3' -> 'c86f7440570f4796babfd44c3d7fd847' move completed: OK
I20240502 14:15:06.189318 16474 rebalancer_tool.cc:1136] tablet b648703b8fc742ecbe1dfd2047d4e68d: 'cd67493f6f0142f896010f2b8a39887d' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:15:06.228854 16474 rebalancer_tool.cc:1136] tablet 5348ba5435c74f509ed77993191ef002: '8b47d4390f8343a493d6c00123785968' -> '7eb4ecb2379b44c0954f2658c4a04350' move completed: OK
I20240502 14:15:06.236784 16474 rebalancer_tool.cc:1136] tablet 3fa51110016d4cafafbe25a1bd213f2e: '20a967bad5504b1e9c6fb720ecdd1625' -> 'd1da7f11f0604f78934354a992bb0f55' move completed: OK
I20240502 14:15:06.251101 16474 rebalancer_tool.cc:1136] tablet 1b34f2a22306459784a1ad7b81cd1683: 'c86f7440570f4796babfd44c3d7fd847' -> 'cd67493f6f0142f896010f2b8a39887d' move completed: OK
I20240502 14:15:06.293207 16474 rebalancer_tool.cc:1136] tablet 8e69d81920d147a38e2542d22ba8352a: '20a967bad5504b1e9c6fb720ecdd1625' -> 'cd67493f6f0142f896010f2b8a39887d' move completed: OK
I20240502 14:15:06.436318 16474 rebalancer_tool.cc:1136] tablet 819e6e83e6064c539cfc73a28c43db5b: 'd1da7f11f0604f78934354a992bb0f55' -> '7eb4ecb2379b44c0954f2658c4a04350' move completed: OK
I20240502 14:15:06.447428 16474 rebalancer_tool.cc:1136] tablet dee785c61c73426fb5665c1334ff372b: 'cd67493f6f0142f896010f2b8a39887d' -> '5297e816f782430d9a01220f0471a882' move completed: OK
I20240502 14:15:06.458416 16474 rebalancer_tool.cc:1136] tablet eb27b3bb29754d7399b0314ada6b1ba7: 'd1da7f11f0604f78934354a992bb0f55' -> '20a967bad5504b1e9c6fb720ecdd1625' move completed: OK
I20240502 14:15:06.471897 16474 rebalancer_tool.cc:1136] tablet 2e51feadc94c48e9bcecffecacbda4b9: '7eb4ecb2379b44c0954f2658c4a04350' -> '5297e816f782430d9a01220f0471a882' move completed: OK
I20240502 14:15:06.561489 16474 rebalancer_tool.cc:1074] tablet b2905b7940fb45c49ef8b18943dae256: 'cd67493f6f0142f896010f2b8a39887d' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:15:06.589202 16474 rebalancer_tool.cc:1074] tablet 9c5329f38f3343baaf43265990d86938: 'cd67493f6f0142f896010f2b8a39887d' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:15:06.624894 16474 rebalancer_tool.cc:1074] tablet 7da975869e82479b91fdda30680e6b3c: '5297e816f782430d9a01220f0471a882' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:15:06.678630 16474 rebalancer_tool.cc:1074] tablet d5975e02205a43d5b93ed8ab51f8a0f8: '7eb4ecb2379b44c0954f2658c4a04350' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:15:06.766343 16474 rebalancer_tool.cc:1074] tablet 6395ce4c30eb4644b0a711de3e60b9c0: 'cd67493f6f0142f896010f2b8a39887d' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:15:07.013520 16474 rebalancer_tool.cc:1074] tablet 652e49046212458280095c016a342f47: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:15:07.079921 16474 rebalancer_tool.cc:1074] tablet 1d607f6e65194399844bc14457fe8f15: '5297e816f782430d9a01220f0471a882' -> '88b77f6edc1c4a72b548bf47c05704f3' move scheduled
I20240502 14:15:08.210184 16474 rebalancer_tool.cc:1136] tablet caa4af5f41674aaa8587158f25e495fb: 'd1da7f11f0604f78934354a992bb0f55' -> 'cd67493f6f0142f896010f2b8a39887d' move completed: OK
I20240502 14:15:08.463006 16474 rebalancer_tool.cc:1136] tablet 0dd07adcbb1f4bd4bcb13bd089c89974: '5297e816f782430d9a01220f0471a882' -> '20a967bad5504b1e9c6fb720ecdd1625' move completed: OK
I20240502 14:15:08.476075 16474 rebalancer_tool.cc:1136] tablet 0e393a7736b14d5483973f9cb89058f8: '88b77f6edc1c4a72b548bf47c05704f3' -> 'c86f7440570f4796babfd44c3d7fd847' move completed: OK
I20240502 14:15:08.485457 16474 rebalancer_tool.cc:1136] tablet 7852539de07744caaf385620c5166861: '88b77f6edc1c4a72b548bf47c05704f3' -> '7eb4ecb2379b44c0954f2658c4a04350' move completed: OK
I20240502 14:15:08.518764 16474 rebalancer_tool.cc:1074] tablet e17e489fe5e04a758d9d9d0259159c64: '20a967bad5504b1e9c6fb720ecdd1625' -> '88b77f6edc1c4a72b548bf47c05704f3' move scheduled
I20240502 14:15:08.558933 16474 rebalancer_tool.cc:1074] tablet 8a98395ed74c4c2eb265a141aec14bd3: '8b47d4390f8343a493d6c00123785968' -> '88b77f6edc1c4a72b548bf47c05704f3' move scheduled
I20240502 14:15:08.666678 16474 rebalancer_tool.cc:1136] tablet 9c5329f38f3343baaf43265990d86938: 'cd67493f6f0142f896010f2b8a39887d' -> 'd1da7f11f0604f78934354a992bb0f55' move completed: OK
I20240502 14:15:08.911671 16474 rebalancer_tool.cc:1136] tablet b2905b7940fb45c49ef8b18943dae256: 'cd67493f6f0142f896010f2b8a39887d' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:15:09.268798 16474 rebalancer_tool.cc:1136] tablet 3d39f694c8554638a3b10ebf7cd01350: '88b77f6edc1c4a72b548bf47c05704f3' -> 'c86f7440570f4796babfd44c3d7fd847' move completed: OK
I20240502 14:15:09.505077 16474 rebalancer_tool.cc:1074] tablet 9a95b9049a344b98be91b9961f3e5bee: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:15:09.553284 16474 rebalancer_tool.cc:1136] tablet 1d607f6e65194399844bc14457fe8f15: '5297e816f782430d9a01220f0471a882' -> '88b77f6edc1c4a72b548bf47c05704f3' move completed: OK
I20240502 14:15:09.561540 16474 rebalancer_tool.cc:1136] tablet 652e49046212458280095c016a342f47: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move completed: OK
I20240502 14:15:09.579695 16474 rebalancer_tool.cc:1136] tablet d5975e02205a43d5b93ed8ab51f8a0f8: '7eb4ecb2379b44c0954f2658c4a04350' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:15:09.702091 16474 rebalancer_tool.cc:1074] tablet 2fc251d4e82e452d989405e121d08dd4: '5297e816f782430d9a01220f0471a882' -> '88b77f6edc1c4a72b548bf47c05704f3' move scheduled
I20240502 14:15:09.808581 16474 rebalancer_tool.cc:1074] tablet c3e9160c46d64fa0a964cc39aec91dfd: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:15:10.326925 16474 rebalancer_tool.cc:1136] tablet e17e489fe5e04a758d9d9d0259159c64: '20a967bad5504b1e9c6fb720ecdd1625' -> '88b77f6edc1c4a72b548bf47c05704f3' move completed: OK
I20240502 14:15:10.494369 16474 rebalancer_tool.cc:1136] tablet 7da975869e82479b91fdda30680e6b3c: '5297e816f782430d9a01220f0471a882' -> '20a967bad5504b1e9c6fb720ecdd1625' move completed: OK
I20240502 14:15:10.565866 16474 rebalancer_tool.cc:1136] tablet 8a98395ed74c4c2eb265a141aec14bd3: '8b47d4390f8343a493d6c00123785968' -> '88b77f6edc1c4a72b548bf47c05704f3' move completed: OK
I20240502 14:15:10.588608 16474 rebalancer_tool.cc:1074] tablet ae45cbb45edf41808686de3418231730: '88b77f6edc1c4a72b548bf47c05704f3' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:15:10.678514 16474 rebalancer_tool.cc:1074] tablet 22705e1aee0947d8996575b93efd3f1f: '88b77f6edc1c4a72b548bf47c05704f3' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:15:11.354264 16474 rebalancer_tool.cc:1136] tablet 2fc251d4e82e452d989405e121d08dd4: '5297e816f782430d9a01220f0471a882' -> '88b77f6edc1c4a72b548bf47c05704f3' move completed: OK
I20240502 14:15:11.491016 16474 rebalancer_tool.cc:1136] tablet 9a95b9049a344b98be91b9961f3e5bee: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move completed: OK
I20240502 14:15:11.541277 16474 rebalancer_tool.cc:1136] tablet 6395ce4c30eb4644b0a711de3e60b9c0: 'cd67493f6f0142f896010f2b8a39887d' -> 'c86f7440570f4796babfd44c3d7fd847' move completed: OK
I20240502 14:15:12.151832 16474 rebalancer_tool.cc:1136] tablet ae45cbb45edf41808686de3418231730: '88b77f6edc1c4a72b548bf47c05704f3' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:15:12.181569 16474 rebalancer_tool.cc:1136] tablet 22705e1aee0947d8996575b93efd3f1f: '88b77f6edc1c4a72b548bf47c05704f3' -> '5297e816f782430d9a01220f0471a882' move completed: OK
I20240502 14:15:12.492106 16474 rebalancer_tool.cc:1136] tablet c3e9160c46d64fa0a964cc39aec91dfd: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move completed: OK
W20240502 14:15:13.416427 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 6cab90f7f1c84cf2b82ee344af1b6617: could not find any suitable replica to move from server c86f7440570f4796babfd44c3d7fd847 to server 7eb4ecb2379b44c0954f2658c4a04350
W20240502 14:15:13.416570 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table fc31add041984706b1347b9ccdaecbaf: could not find any suitable replica to move from server c86f7440570f4796babfd44c3d7fd847 to server 8b47d4390f8343a493d6c00123785968
W20240502 14:15:13.416697 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 35b2e9aa3c11478e9023ef811af72c1a: could not find any suitable replica to move from server d1da7f11f0604f78934354a992bb0f55 to server 88b77f6edc1c4a72b548bf47c05704f3
W20240502 14:15:13.416976 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 8f0281d7a3a049329cb3130e1f8d0c04: could not find any suitable replica to move from server d1da7f11f0604f78934354a992bb0f55 to server cd67493f6f0142f896010f2b8a39887d
W20240502 14:15:13.417130 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 35b2e9aa3c11478e9023ef811af72c1a: could not find any suitable replica to move from server 5297e816f782430d9a01220f0471a882 to server c86f7440570f4796babfd44c3d7fd847
W20240502 14:15:13.417258 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 9dc6b07ff15645ab9222cc55c5a4a00b: could not find any suitable replica to move from server c86f7440570f4796babfd44c3d7fd847 to server 7eb4ecb2379b44c0954f2658c4a04350
I20240502 14:15:13.440874 16474 rebalancer_tool.cc:1074] tablet 2363a23cdde54893ae807ef3960dd032: '8b47d4390f8343a493d6c00123785968' -> '88b77f6edc1c4a72b548bf47c05704f3' move scheduled
I20240502 14:15:13.480121 16474 rebalancer_tool.cc:1074] tablet 154fe82369ef4d2bb38e8f14637463bb: 'c86f7440570f4796babfd44c3d7fd847' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:15:13.554072 16474 rebalancer_tool.cc:1074] tablet c28e61290d744d4c9409947f9e41706e: 'd1da7f11f0604f78934354a992bb0f55' -> '7eb4ecb2379b44c0954f2658c4a04350' move scheduled
I20240502 14:15:13.590559 16474 rebalancer_tool.cc:1074] tablet 2cb770f077c84cb99985ef59253f133b: '8b47d4390f8343a493d6c00123785968' -> 'cd67493f6f0142f896010f2b8a39887d' move scheduled
I20240502 14:15:13.654994 16474 rebalancer_tool.cc:1074] tablet a1c0b2fdd1d64bac8100d797c86fd8b2: '5297e816f782430d9a01220f0471a882' -> '88b77f6edc1c4a72b548bf47c05704f3' move scheduled
I20240502 14:15:13.721405 16474 rebalancer_tool.cc:1074] tablet 7a740ff5e0964efcb1fd7dd54dc4094c: 'c86f7440570f4796babfd44c3d7fd847' -> '7eb4ecb2379b44c0954f2658c4a04350' move scheduled
I20240502 14:15:13.778046 16474 rebalancer_tool.cc:1074] tablet 16c817b864c4425eaa85ac4b5aff59fa: '20a967bad5504b1e9c6fb720ecdd1625' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:15:13.853425 16474 rebalancer_tool.cc:1074] tablet cee69dfc4ded45de81cfd8b06c5f2f2f: '88b77f6edc1c4a72b548bf47c05704f3' -> 'cd67493f6f0142f896010f2b8a39887d' move scheduled
I20240502 14:15:13.897938 16474 rebalancer_tool.cc:1074] tablet 1b34f2a22306459784a1ad7b81cd1683: '5297e816f782430d9a01220f0471a882' -> '7eb4ecb2379b44c0954f2658c4a04350' move scheduled
I20240502 14:15:14.162534 16474 rebalancer_tool.cc:1074] tablet eb472259eca84c31b2ff9f33f7e61177: '8b47d4390f8343a493d6c00123785968' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:15:14.599148 16474 rebalancer_tool.cc:1074] tablet 1c251cfdbec94259be5afbb1b7dbaa38: 'd1da7f11f0604f78934354a992bb0f55' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:15:14.710211 16474 rebalancer_tool.cc:1074] tablet caac1154570d4420ae8764ca18667f29: 'cd67493f6f0142f896010f2b8a39887d' -> '7eb4ecb2379b44c0954f2658c4a04350' move scheduled
I20240502 14:15:14.889369 16474 rebalancer_tool.cc:1074] tablet db2bc203d47c4ca1b3329745a0ff7fe1: 'd1da7f11f0604f78934354a992bb0f55' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:15:15.765982 16474 rebalancer_tool.cc:1074] tablet c78403038b614e6db2afb617328f6411: '8b47d4390f8343a493d6c00123785968' -> '88b77f6edc1c4a72b548bf47c05704f3' move scheduled
I20240502 14:15:16.266393 16474 rebalancer_tool.cc:1074] tablet e3be42ac139741d19dd872ab551bebae: '7eb4ecb2379b44c0954f2658c4a04350' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:15:16.448392 16474 rebalancer_tool.cc:1074] tablet 9c5329f38f3343baaf43265990d86938: '20a967bad5504b1e9c6fb720ecdd1625' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:15:16.546866 16474 rebalancer_tool.cc:1074] tablet 76c85c912ab74c32bfed536ea38d23f6: 'cd67493f6f0142f896010f2b8a39887d' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:15:16.575678 16474 rebalancer_tool.cc:1074] tablet 76b46f71180f4269bee53f206e9c2665: '8b47d4390f8343a493d6c00123785968' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:15:16.604389 16474 rebalancer_tool.cc:1074] tablet 5c4f5c4167ac4c30af3aacb7f7cd099b: '5297e816f782430d9a01220f0471a882' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:15:16.756522 16474 rebalancer_tool.cc:1136] tablet 1c251cfdbec94259be5afbb1b7dbaa38: 'd1da7f11f0604f78934354a992bb0f55' -> '20a967bad5504b1e9c6fb720ecdd1625' move completed: OK
I20240502 14:15:16.781739 16474 rebalancer_tool.cc:1136] tablet eb472259eca84c31b2ff9f33f7e61177: '8b47d4390f8343a493d6c00123785968' -> 'c86f7440570f4796babfd44c3d7fd847' move completed: OK
I20240502 14:15:16.801195 16474 rebalancer_tool.cc:1136] tablet 1b34f2a22306459784a1ad7b81cd1683: '5297e816f782430d9a01220f0471a882' -> '7eb4ecb2379b44c0954f2658c4a04350' move completed: OK
I20240502 14:15:17.476559 16474 rebalancer_tool.cc:1136] tablet cee69dfc4ded45de81cfd8b06c5f2f2f: '88b77f6edc1c4a72b548bf47c05704f3' -> 'cd67493f6f0142f896010f2b8a39887d' move completed: OK
I20240502 14:15:17.583544 16474 rebalancer_tool.cc:1136] tablet 16c817b864c4425eaa85ac4b5aff59fa: '20a967bad5504b1e9c6fb720ecdd1625' -> 'd1da7f11f0604f78934354a992bb0f55' move completed: OK
I20240502 14:15:17.653209 16474 rebalancer_tool.cc:1136] tablet 7a740ff5e0964efcb1fd7dd54dc4094c: 'c86f7440570f4796babfd44c3d7fd847' -> '7eb4ecb2379b44c0954f2658c4a04350' move completed: OK
I20240502 14:15:17.750259 16474 rebalancer_tool.cc:1136] tablet a1c0b2fdd1d64bac8100d797c86fd8b2: '5297e816f782430d9a01220f0471a882' -> '88b77f6edc1c4a72b548bf47c05704f3' move completed: OK
I20240502 14:15:17.805482 16474 rebalancer_tool.cc:1136] tablet 2cb770f077c84cb99985ef59253f133b: '8b47d4390f8343a493d6c00123785968' -> 'cd67493f6f0142f896010f2b8a39887d' move completed: OK
I20240502 14:15:17.815719 16474 rebalancer_tool.cc:1136] tablet e3be42ac139741d19dd872ab551bebae: '7eb4ecb2379b44c0954f2658c4a04350' -> 'c86f7440570f4796babfd44c3d7fd847' move completed: OK
I20240502 14:15:17.826032 16474 rebalancer_tool.cc:1136] tablet 154fe82369ef4d2bb38e8f14637463bb: 'c86f7440570f4796babfd44c3d7fd847' -> '20a967bad5504b1e9c6fb720ecdd1625' move completed: OK
I20240502 14:15:17.845422 16474 rebalancer_tool.cc:1136] tablet c28e61290d744d4c9409947f9e41706e: 'd1da7f11f0604f78934354a992bb0f55' -> '7eb4ecb2379b44c0954f2658c4a04350' move completed: OK
I20240502 14:15:17.865267 16474 rebalancer_tool.cc:1136] tablet 2363a23cdde54893ae807ef3960dd032: '8b47d4390f8343a493d6c00123785968' -> '88b77f6edc1c4a72b548bf47c05704f3' move completed: OK
I20240502 14:15:17.965785 16474 rebalancer_tool.cc:1074] tablet 861f9171cb654ec889ddf824f2bb6e92: 'c86f7440570f4796babfd44c3d7fd847' -> '88b77f6edc1c4a72b548bf47c05704f3' move scheduled
I20240502 14:15:18.071539 16474 rebalancer_tool.cc:1074] tablet 0e393a7736b14d5483973f9cb89058f8: '7eb4ecb2379b44c0954f2658c4a04350' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:15:18.155697 16474 rebalancer_tool.cc:1074] tablet e7b93f8449ef4e6fb0a7e315130da024: 'd1da7f11f0604f78934354a992bb0f55' -> 'cd67493f6f0142f896010f2b8a39887d' move scheduled
I20240502 14:15:18.253695 16474 rebalancer_tool.cc:1074] tablet 0854e9f4851e400f9af18181a71ae8f7: '7eb4ecb2379b44c0954f2658c4a04350' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:15:18.656874 16474 rebalancer_tool.cc:1074] tablet 27857198d4ec430ebee897c90141c58d: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move scheduled
I20240502 14:15:18.932297 16474 rebalancer_tool.cc:1074] tablet fae5b25ec9694cea87381d27f4d46384: '20a967bad5504b1e9c6fb720ecdd1625' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:15:19.074146 16474 rebalancer_tool.cc:1074] tablet c5652dfa60d84803abbac9201e5874db: 'd1da7f11f0604f78934354a992bb0f55' -> 'cd67493f6f0142f896010f2b8a39887d' move scheduled
I20240502 14:15:19.238920 16474 rebalancer_tool.cc:1074] tablet cd06e00fd93340099df5e020716033a9: '8b47d4390f8343a493d6c00123785968' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:15:19.852705 16474 rebalancer_tool.cc:1136] tablet e7b93f8449ef4e6fb0a7e315130da024: 'd1da7f11f0604f78934354a992bb0f55' -> 'cd67493f6f0142f896010f2b8a39887d' move completed: OK
I20240502 14:15:19.862784 16474 rebalancer_tool.cc:1136] tablet 5c4f5c4167ac4c30af3aacb7f7cd099b: '5297e816f782430d9a01220f0471a882' -> '20a967bad5504b1e9c6fb720ecdd1625' move completed: OK
I20240502 14:15:20.359475 16474 rebalancer_tool.cc:1136] tablet 9c5329f38f3343baaf43265990d86938: '20a967bad5504b1e9c6fb720ecdd1625' -> '5297e816f782430d9a01220f0471a882' move completed: OK
I20240502 14:15:20.429543 16474 rebalancer_tool.cc:1136] tablet 861f9171cb654ec889ddf824f2bb6e92: 'c86f7440570f4796babfd44c3d7fd847' -> '88b77f6edc1c4a72b548bf47c05704f3' move completed: OK
I20240502 14:15:20.491834 16474 rebalancer_tool.cc:1136] tablet c78403038b614e6db2afb617328f6411: '8b47d4390f8343a493d6c00123785968' -> '88b77f6edc1c4a72b548bf47c05704f3' move completed: OK
I20240502 14:15:20.531098 16474 rebalancer_tool.cc:1136] tablet 0e393a7736b14d5483973f9cb89058f8: '7eb4ecb2379b44c0954f2658c4a04350' -> '20a967bad5504b1e9c6fb720ecdd1625' move completed: OK
I20240502 14:15:20.543133 16474 rebalancer_tool.cc:1136] tablet caac1154570d4420ae8764ca18667f29: 'cd67493f6f0142f896010f2b8a39887d' -> '7eb4ecb2379b44c0954f2658c4a04350' move completed: OK
I20240502 14:15:20.559510 16474 rebalancer_tool.cc:1136] tablet db2bc203d47c4ca1b3329745a0ff7fe1: 'd1da7f11f0604f78934354a992bb0f55' -> '5297e816f782430d9a01220f0471a882' move completed: OK
I20240502 14:15:20.586140 16474 rebalancer_tool.cc:1136] tablet 76b46f71180f4269bee53f206e9c2665: '8b47d4390f8343a493d6c00123785968' -> 'c86f7440570f4796babfd44c3d7fd847' move completed: OK
I20240502 14:15:20.724748 16474 rebalancer_tool.cc:1074] tablet 524a9a50c4e843a1ad20f071c83b5aec: 'd1da7f11f0604f78934354a992bb0f55' -> '7eb4ecb2379b44c0954f2658c4a04350' move scheduled
I20240502 14:15:20.782882 16474 rebalancer_tool.cc:1074] tablet 990da252c05c45d8b7a3e47288f1ab61: '8b47d4390f8343a493d6c00123785968' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:15:20.843530 16474 rebalancer_tool.cc:1074] tablet e95022a368ec47949d1003c39a7fe433: 'd1da7f11f0604f78934354a992bb0f55' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:15:20.881287 16474 rebalancer_tool.cc:1074] tablet ba8299c5f722440e92803100e7240723: '8b47d4390f8343a493d6c00123785968' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:15:20.974370 16474 rebalancer_tool.cc:1136] tablet cd06e00fd93340099df5e020716033a9: '8b47d4390f8343a493d6c00123785968' -> '20a967bad5504b1e9c6fb720ecdd1625' move completed: OK
I20240502 14:15:20.984153 16474 rebalancer_tool.cc:1136] tablet fae5b25ec9694cea87381d27f4d46384: '20a967bad5504b1e9c6fb720ecdd1625' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:15:20.991788 16474 rebalancer_tool.cc:1136] tablet 0854e9f4851e400f9af18181a71ae8f7: '7eb4ecb2379b44c0954f2658c4a04350' -> '8b47d4390f8343a493d6c00123785968' move completed: OK
I20240502 14:15:20.999279 16474 rebalancer_tool.cc:1136] tablet 27857198d4ec430ebee897c90141c58d: '88b77f6edc1c4a72b548bf47c05704f3' -> 'd1da7f11f0604f78934354a992bb0f55' move completed: OK
I20240502 14:15:21.016752 16474 rebalancer_tool.cc:1136] tablet c5652dfa60d84803abbac9201e5874db: 'd1da7f11f0604f78934354a992bb0f55' -> 'cd67493f6f0142f896010f2b8a39887d' move completed: OK
I20240502 14:15:21.467542 16474 rebalancer_tool.cc:1136] tablet 76c85c912ab74c32bfed536ea38d23f6: 'cd67493f6f0142f896010f2b8a39887d' -> 'd1da7f11f0604f78934354a992bb0f55' move completed: OK
I20240502 14:15:22.486953 16474 rebalancer_tool.cc:1136] tablet 990da252c05c45d8b7a3e47288f1ab61: '8b47d4390f8343a493d6c00123785968' -> '20a967bad5504b1e9c6fb720ecdd1625' move completed: OK
I20240502 14:15:22.551086 16474 rebalancer_tool.cc:1136] tablet e95022a368ec47949d1003c39a7fe433: 'd1da7f11f0604f78934354a992bb0f55' -> '20a967bad5504b1e9c6fb720ecdd1625' move completed: OK
I20240502 14:15:22.560885 16474 rebalancer_tool.cc:1136] tablet ba8299c5f722440e92803100e7240723: '8b47d4390f8343a493d6c00123785968' -> '20a967bad5504b1e9c6fb720ecdd1625' move completed: OK
I20240502 14:15:23.673311 16474 rebalancer_tool.cc:1136] tablet 524a9a50c4e843a1ad20f071c83b5aec: 'd1da7f11f0604f78934354a992bb0f55' -> '7eb4ecb2379b44c0954f2658c4a04350' move completed: OK
W20240502 14:15:24.340119 16474 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 176de60994624cffa95f6e64915365f1: could not find any suitable replica to move from server 5297e816f782430d9a01220f0471a882 to server d1da7f11f0604f78934354a992bb0f55
I20240502 14:15:24.368604 16474 rebalancer_tool.cc:1074] tablet 57b8b76eff294d42a45e3c9240ff23c8: '8b47d4390f8343a493d6c00123785968' -> 'cd67493f6f0142f896010f2b8a39887d' move scheduled
I20240502 14:15:24.411486 16474 rebalancer_tool.cc:1074] tablet 939a92466f61423187582d84ab10c0f6: '5297e816f782430d9a01220f0471a882' -> '7eb4ecb2379b44c0954f2658c4a04350' move scheduled
I20240502 14:15:24.457002 16474 rebalancer_tool.cc:1074] tablet 61c065054cdb456da06172650abf0fae: 'c86f7440570f4796babfd44c3d7fd847' -> '20a967bad5504b1e9c6fb720ecdd1625' move scheduled
I20240502 14:15:24.490767 16474 rebalancer_tool.cc:1074] tablet ed5f3b784fa44b09974bc9caf17049af: 'd1da7f11f0604f78934354a992bb0f55' -> '88b77f6edc1c4a72b548bf47c05704f3' move scheduled
I20240502 14:15:24.532496 16474 rebalancer_tool.cc:1074] tablet d1e9c049ffe74f20b569b9ac8f8a7592: '8b47d4390f8343a493d6c00123785968' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:15:24.561666 16474 rebalancer_tool.cc:1074] tablet eafe2272cca9440ba7060f64106a73cb: 'cd67493f6f0142f896010f2b8a39887d' -> 'c86f7440570f4796babfd44c3d7fd847' move scheduled
I20240502 14:15:24.608605 16474 rebalancer_tool.cc:1074] tablet 0a2fc48c889b481e9b4d249819a196a0: '7eb4ecb2379b44c0954f2658c4a04350' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:15:24.642263 16474 rebalancer_tool.cc:1074] tablet b648703b8fc742ecbe1dfd2047d4e68d: 'd1da7f11f0604f78934354a992bb0f55' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:15:24.676748 16474 rebalancer_tool.cc:1074] tablet 1b6760f0772449088e93bc1eab5e2ab3: '88b77f6edc1c4a72b548bf47c05704f3' -> 'cd67493f6f0142f896010f2b8a39887d' move scheduled
I20240502 14:15:24.720925 16474 rebalancer_tool.cc:1074] tablet a1c0b2fdd1d64bac8100d797c86fd8b2: 'c86f7440570f4796babfd44c3d7fd847' -> '7eb4ecb2379b44c0954f2658c4a04350' move scheduled
I20240502 14:15:24.745792 16474 rebalancer_tool.cc:1074] tablet df55a71945d34af0affa6eccbdf84dae: 'd1da7f11f0604f78934354a992bb0f55' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:15:24.777415 16474 rebalancer_tool.cc:1074] tablet e93e561ce7ec4e6fa7e211fd1e4fba56: '88b77f6edc1c4a72b548bf47c05704f3' -> '5297e816f782430d9a01220f0471a882' move scheduled
I20240502 14:15:24.844142 16474 rebalancer_tool.cc:1074] tablet 07012e715ded463db5edbb6019ee69cf: '5297e816f782430d9a01220f0471a882' -> 'cd67493f6f0142f896010f2b8a39887d' move scheduled
I20240502 14:15:24.926931 16474 rebalancer_tool.cc:1074] tablet 8a98395ed74c4c2eb265a141aec14bd3: 'c86f7440570f4796babfd44c3d7fd847' -> '8b47d4390f8343a493d6c00123785968' move scheduled
I20240502 14:15:25.732415 16474 rebalancer_tool.cc:1123] tablet e93e561ce7ec4e6fa7e211fd1e4fba56: '88b77f6edc1c4a72b548bf47c05704f3' -> '5297e816f782430d9a01220f0471a882' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: CW20240502 14:15:27.209648 13516 consensus_peers.cc:485] T caac1154570d4420ae8764ca18667f29 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.210038 13121 consensus_peers.cc:462] T 0a2fc48c889b481e9b4d249819a196a0 P 20a967bad5504b1e9c6fb720ecdd1625 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
onnection refused (error 111)
I20240502 14:15:25.754736 16474 rebalancer_tool.cc:1123] tablet df55a71945d34af0affa6eccbdf84dae: 'd1da7f11f0604f78934354a992bb0f55' -> '8b47d4390f8343a493d6c00123785968' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:25.767163 16474 rebalancer_tool.cc:1123] tablet a1c0b2fdd1d64bac8100d797c86fd8b2: 'c86f7440570f4796babfd44c3d7fd847' -> '7eb4ecb2379b44c0954f2658c4a04350' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:25.832245 16474 rebalancer_tool.cc:1123] tablet 07012e715ded463db5edbb6019ee69cf: '5297e816f782430d9a01220f0471a882' -> 'cd67493f6f0142f896010f2b8a39887d' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
W20240502 14:15:25.857928 21906 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:15:25.846997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.196.199:33857 (local address 127.0.0.1:59082)
0502 14:15:25.847478 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0502 14:15:25.847496 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0502 14:15:25.847605 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:15:25.856669 (+ 9064us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":249,"threads_started":1}
I20240502 14:15:25.863180 16474 rebalancer_tool.cc:1123] tablet eafe2272cca9440ba7060f64106a73cb: 'cd67493f6f0142f896010f2b8a39887d' -> 'c86f7440570f4796babfd44c3d7fd847' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:15:26.031152 16474 rebalancer_tool.cc:1123] tablet 61c065054cdb456da06172650abf0fae: 'c86f7440570f4796babfd44c3d7fd847' -> '20a967bad5504b1e9c6fb720ecdd1625' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.051638 16474 rebalancer_tool.cc:1136] tablet 939a92466f61423187582d84ab10c0f6: '5297e816f782430d9a01220f0471a882' -> '7eb4ecb2379b44c0954f2658c4a04350' move completed: OK
I20240502 14:15:26.056790 16474 rebalancer_tool.cc:1123] tablet 57b8b76eff294d42a45e3c9240ff23c8: '8b47d4390f8343a493d6c00123785968' -> 'cd67493f6f0142f896010f2b8a39887d' move is abandoned: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:26.060386 16474 rebalancer_tool.cc:1084] tablet c28e61290d744d4c9409947f9e41706e: '5297e816f782430d9a01220f0471a882' -> '88b77f6edc1c4a72b548bf47c05704f3' move ignored: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:26.060535 16474 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): unacceptable health status UNAVAILABLE
I20240502 14:15:27.214459 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 13105
I20240502 14:15:27.223080 13186 raft_consensus.cc:1261] T 7852539de07744caaf385620c5166861 P 20a967bad5504b1e9c6fb720ecdd1625 [term 3 FOLLOWER]: Refusing update from remote peer 7eb4ecb2379b44c0954f2658c4a04350: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 14:15:27.223743 21931 consensus_queue.cc:1035] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:15:27.225064 13228 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:27.232004 13578 raft_consensus.cc:1261] T 1b34f2a22306459784a1ad7b81cd1683 P 7eb4ecb2379b44c0954f2658c4a04350 [term 2 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240502 14:15:27.232695 21707 consensus_queue.cc:1035] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240502 14:15:27.245992 13252 consensus_peers.cc:485] T 1b34f2a22306459784a1ad7b81cd1683 P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.267845 13031 catalog_manager.cc:4573] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 939a92466f61423187582d84ab10c0f6: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:27.319134 13253 consensus_peers.cc:485] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.412436 15957 consensus_peers.cc:485] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.426568 13516 consensus_peers.cc:485] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:15:27.447515 13318 raft_consensus.cc:1261] T 2cb770f077c84cb99985ef59253f133b P cd67493f6f0142f896010f2b8a39887d [term 2 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240502 14:15:27.447988 21870 consensus_queue.cc:1035] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240502 14:15:27.455297 15957 consensus_peers.cc:485] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.465082 15956 consensus_peers.cc:485] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:15:27.545397 13447 raft_consensus.cc:1261] T 9a42ded434514cf28e578eb0046dc5f5 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Refusing update from remote peer cd67493f6f0142f896010f2b8a39887d: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:15:27.545886 21707 consensus_queue.cc:1035] T 9a42ded434514cf28e578eb0046dc5f5 P cd67493f6f0142f896010f2b8a39887d [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:27.587635 13187 raft_consensus.cc:1261] T c871ce05c0804c2fbb3cb06049e6d56a P 20a967bad5504b1e9c6fb720ecdd1625 [term 2 FOLLOWER]: Refusing update from remote peer d1da7f11f0604f78934354a992bb0f55: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240502 14:15:27.592130 22038 consensus_queue.cc:1035] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240502 14:15:27.602085 13318 raft_consensus.cc:1261] T 0e393a7736b14d5483973f9cb89058f8 P cd67493f6f0142f896010f2b8a39887d [term 3 FOLLOWER]: Refusing update from remote peer 20a967bad5504b1e9c6fb720ecdd1625: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 3 index: 11. (index mismatch)
I20240502 14:15:27.604455 21587 consensus_queue.cc:1035] T 0e393a7736b14d5483973f9cb89058f8 P 20a967bad5504b1e9c6fb720ecdd1625 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240502 14:15:27.616189 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 13105
W20240502 14:15:27.633438 13515 consensus_peers.cc:485] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.662114 13516 consensus_peers.cc:462] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:27.669929 15955 connection.cc:535] client connection to 127.6.196.193:36687 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:27.669970 13251 connection.cc:535] client connection to 127.6.196.193:36687 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:27.670809 13254 connection.cc:535] server connection from 127.6.196.193:32917 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:27.670909 13516 connection.cc:535] server connection from 127.6.196.193:50547 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:27.671106 13382 connection.cc:535] client connection to 127.6.196.193:36687 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:27.671595 13514 connection.cc:535] client connection to 127.6.196.193:36687 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:15:27.671797 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 13236
W20240502 14:15:27.672097 13382 consensus_peers.cc:485] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.672768 22030 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:15:27.670627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.196.193:36687 (local address 127.6.196.197:35661)
0502 14:15:27.670861 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0502 14:15:27.670878 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0502 14:15:27.670992 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0502 14:15:27.672518 (+ 1526us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":138}
W20240502 14:15:27.673094 15955 consensus_peers.cc:485] T c871ce05c0804c2fbb3cb06049e6d56a P d1da7f11f0604f78934354a992bb0f55 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.673270 13514 consensus_peers.cc:485] T 7852539de07744caaf385620c5166861 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.673939 13382 consensus_peers.cc:485] T 8a3b25a9e8b049a88bb999d9362e98da P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.674794 21848 negotiation.cc:337] Failed RPC negotiation. Trace:
0502 14:15:27.673959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.196.193:36687 (local address 127.6.196.194:57009)
0502 14:15:27.674060 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0502 14:15:27.674143 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":31}
W20240502 14:15:27.675745 13251 consensus_peers.cc:485] T 5c4f5c4167ac4c30af3aacb7f7cd099b P cd67493f6f0142f896010f2b8a39887d -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.675845 13251 consensus_peers.cc:485] T ba8299c5f722440e92803100e7240723 P cd67493f6f0142f896010f2b8a39887d -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.675905 13251 consensus_peers.cc:485] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.675959 13251 consensus_peers.cc:485] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.676030 13251 consensus_peers.cc:485] T 939a92466f61423187582d84ab10c0f6 P cd67493f6f0142f896010f2b8a39887d -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.676105 13251 consensus_peers.cc:485] T ae45cbb45edf41808686de3418231730 P cd67493f6f0142f896010f2b8a39887d -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:27.679453 13359 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:15:27.717036 15956 consensus_peers.cc:462] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:27.724752 15955 consensus_peers.cc:485] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.753175 13382 consensus_peers.cc:485] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.768010 13382 consensus_peers.cc:485] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.780347 13515 consensus_peers.cc:485] T 16c817b864c4425eaa85ac4b5aff59fa P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.780481 13383 consensus_peers.cc:485] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.792999 13514 consensus_peers.cc:485] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.800101 15957 consensus_peers.cc:485] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.810099 15955 consensus_peers.cc:485] T 1e2b8c72c7ba4be58f31ac073e41f2a8 P d1da7f11f0604f78934354a992bb0f55 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.818207 15956 consensus_peers.cc:485] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.867650 15955 consensus_peers.cc:485] T 551bf8a880ec4157870bc449c0c54cf9 P d1da7f11f0604f78934354a992bb0f55 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.905369 15957 consensus_peers.cc:485] T dbe34cac4e774d3c9f57bc1d64569175 P d1da7f11f0604f78934354a992bb0f55 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.909870 13516 consensus_peers.cc:485] T 80040c60b029481285c50171bcc5ce35 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.917975 13384 consensus_peers.cc:485] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.924400 13514 consensus_peers.cc:485] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.929817 15956 consensus_peers.cc:485] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.936477 15956 consensus_peers.cc:485] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.939415 13515 consensus_peers.cc:485] T 9ad2eb265e0f469787eb1699e339e0ed P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.951032 15955 consensus_peers.cc:485] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.951866 13515 consensus_peers.cc:485] T 27857198d4ec430ebee897c90141c58d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.957543 13515 consensus_peers.cc:485] T 16d6faae2add42c5bd4af6018a13b841 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.961823 13514 consensus_peers.cc:485] T 766f50e1e5a44c8ba211a446bcf86705 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.982059 15956 consensus_peers.cc:485] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.983532 13382 consensus_peers.cc:485] T ab51b049c3674b629b55ed29df5b1fab P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:27.995577 13252 consensus_peers.cc:485] T cd06e00fd93340099df5e020716033a9 P cd67493f6f0142f896010f2b8a39887d -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.995774 13252 consensus_peers.cc:485] T 21ac2a9fa2154a0bbbe15aeaf178ea1a P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:27.998862 15956 consensus_peers.cc:485] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.000469 13252 consensus_peers.cc:485] T 1d607f6e65194399844bc14457fe8f15 P cd67493f6f0142f896010f2b8a39887d -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.001022 13252 consensus_peers.cc:485] T 20f045499e874907bcde4318796580be P cd67493f6f0142f896010f2b8a39887d -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.001240 13252 consensus_peers.cc:485] T 430afdc079aa4c2b88be1e2a328a64bb P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.001410 13252 consensus_peers.cc:485] T d5975e02205a43d5b93ed8ab51f8a0f8 P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.001561 13252 consensus_peers.cc:485] T ba519bb3fb534e7fb6106f402e27446d P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.001691 13252 consensus_peers.cc:485] T 0854e9f4851e400f9af18181a71ae8f7 P cd67493f6f0142f896010f2b8a39887d -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:15:28.002955 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 13236
W20240502 14:15:28.004735 13383 consensus_peers.cc:485] T e3be42ac139741d19dd872ab551bebae P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.008589 13383 consensus_peers.cc:485] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.012156 15956 consensus_peers.cc:485] T 748ccda9cdc94ebaaa432562e20f9fa3 P d1da7f11f0604f78934354a992bb0f55 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.017870 13384 consensus_peers.cc:485] T 94ef25b3a2cd4bc5a53385045336f45a P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.033704 13383 consensus_peers.cc:485] T 0dd07adcbb1f4bd4bcb13bd089c89974 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:15:28.041289 21876 raft_consensus.cc:482] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:28.041504 21876 raft_consensus.cc:504] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:28.043920 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e561ce7ec4e6fa7e211fd1e4fba56" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55" is_pre_election: true
I20240502 14:15:28.044145 16018 raft_consensus.cc:2454] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 2.
W20240502 14:15:28.044468 13383 leader_election.cc:334] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:28.044696 13384 leader_election.cc:302] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, d1da7f11f0604f78934354a992bb0f55; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:28.043578 21876 leader_election.cc:288] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:28.045157 22100 raft_consensus.cc:2781] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240502 14:15:28.045265 22100 raft_consensus.cc:482] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:28.045367 22100 raft_consensus.cc:3037] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:15:28.046766 15956 consensus_peers.cc:485] T 7da975869e82479b91fdda30680e6b3c P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:15:28.047791 22100 raft_consensus.cc:504] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
I20240502 14:15:28.048535 22100 leader_election.cc:288] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 3 election: Requested vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), d1da7f11f0604f78934354a992bb0f55 (127.6.196.197:43695)
I20240502 14:15:28.049019 16018 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e93e561ce7ec4e6fa7e211fd1e4fba56" candidate_uuid: "88b77f6edc1c4a72b548bf47c05704f3" candidate_term: 3 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "d1da7f11f0604f78934354a992bb0f55"
I20240502 14:15:28.049186 16018 raft_consensus.cc:3037] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Advancing to term 3
W20240502 14:15:28.049798 13383 leader_election.cc:334] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:28.051360 13514 consensus_peers.cc:485] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:28.051748 16018 raft_consensus.cc:2454] T e93e561ce7ec4e6fa7e211fd1e4fba56 P d1da7f11f0604f78934354a992bb0f55 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88b77f6edc1c4a72b548bf47c05704f3 in term 3.
I20240502 14:15:28.052124 13384 leader_election.cc:302] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3, d1da7f11f0604f78934354a992bb0f55; no voters: 8b47d4390f8343a493d6c00123785968
I20240502 14:15:28.052383 22100 raft_consensus.cc:2781] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:28.052520 22100 raft_consensus.cc:686] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [term 3 LEADER]: Becoming Leader. State: Replica: 88b77f6edc1c4a72b548bf47c05704f3, State: Running, Role: LEADER
I20240502 14:15:28.052793 22100 consensus_queue.cc:237] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } }
W20240502 14:15:28.052989 13516 connection.cc:535] server connection from 127.6.196.194:35821 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:28.053139 13516 connection.cc:535] client connection to 127.6.196.194:33193 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:15:28.053973 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 13367
W20240502 14:15:28.054114 13516 consensus_peers.cc:485] T 4e05e25ef22e40d380ab7f83437e1f4b P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.194:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.054219 13516 consensus_peers.cc:485] T cee69dfc4ded45de81cfd8b06c5f2f2f P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.194:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.054366 15957 connection.cc:535] server connection from 127.6.196.194:57811 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:15:28.057693 13044 catalog_manager.cc:5484] T e93e561ce7ec4e6fa7e211fd1e4fba56 P 88b77f6edc1c4a72b548bf47c05704f3 reported cstate change: term changed from 1 to 3, leader changed from 8b47d4390f8343a493d6c00123785968 (127.6.196.200) to 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195). New cstate: current_term: 3 leader_uuid: "88b77f6edc1c4a72b548bf47c05704f3" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: NON_VOTER last_known_addr { host: "127.6.196.199" port: 33857 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:28.060331 13490 generic_service.cc:196] Checking for leaks (request via RPC)
W20240502 14:15:28.065028 15957 consensus_peers.cc:485] T 2cb770f077c84cb99985ef59253f133b P d1da7f11f0604f78934354a992bb0f55 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.194:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.072692 15955 consensus_peers.cc:485] T 9535cdbc4294429593012b088528c449 P d1da7f11f0604f78934354a992bb0f55 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.113619 15955 consensus_peers.cc:485] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.131084 15956 consensus_peers.cc:485] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.165509 13515 consensus_peers.cc:485] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.167297 15957 consensus_peers.cc:485] T 8e3da1778819497bb3bd56f9ba451046 P d1da7f11f0604f78934354a992bb0f55 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.194:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.167307 15955 consensus_peers.cc:485] T 154fe82369ef4d2bb38e8f14637463bb P d1da7f11f0604f78934354a992bb0f55 -> Peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Couldn't send request to peer 20a967bad5504b1e9c6fb720ecdd1625. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.171860 15956 consensus_peers.cc:485] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.177901 13516 consensus_peers.cc:462] T b648703b8fc742ecbe1dfd2047d4e68d P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:28.190586 15956 consensus_peers.cc:462] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:28.197157 15956 consensus_peers.cc:485] T 5661c0f4fe074dd28e81d195d6607960 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.291414 13515 consensus_peers.cc:485] T 652e49046212458280095c016a342f47 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.294373 13515 consensus_peers.cc:485] T d1e9c049ffe74f20b569b9ac8f8a7592 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240502 14:15:28.329981 22090 raft_consensus.cc:482] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:28.330179 22090 raft_consensus.cc:504] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: NON_VOTER last_known_addr { host: "127.6.196.194" port: 33193 } attrs { promote: true } }
I20240502 14:15:28.330808 22090 leader_election.cc:288] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
W20240502 14:15:28.332276 13515 leader_election.cc:334] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:28.332393 13516 leader_election.cc:334] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:28.332480 13516 leader_election.cc:302] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7eb4ecb2379b44c0954f2658c4a04350; no voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:28.332690 22090 raft_consensus.cc:2726] T 57b8b76eff294d42a45e3c9240ff23c8 P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:28.336648 13031 catalog_manager.cc:4573] TS 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 939a92466f61423187582d84ab10c0f6: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:28.365581 22114 raft_consensus.cc:482] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:28.365733 22114 raft_consensus.cc:504] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "88b77f6edc1c4a72b548bf47c05704f3" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 46303 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } }
I20240502 14:15:28.366403 22114 leader_election.cc:288] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
W20240502 14:15:28.369277 13383 leader_election.cc:334] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:28.369442 13383 consensus_peers.cc:485] T 274c6eb091a742089999cc39ad6dd76d P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Couldn't send request to peer 8b47d4390f8343a493d6c00123785968. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.373262 13384 leader_election.cc:334] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:28.373420 13384 leader_election.cc:302] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 88b77f6edc1c4a72b548bf47c05704f3; no voters: 5297e816f782430d9a01220f0471a882, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:28.373693 22100 raft_consensus.cc:2726] T a110350f069e45869eea48967f77e5fc P 88b77f6edc1c4a72b548bf47c05704f3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:28.374107 13384 consensus_peers.cc:485] T c5652dfa60d84803abbac9201e5874db P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.194:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.374248 13384 consensus_peers.cc:485] T 536c22c444344bb485d3e522f0f77154 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.194:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.374362 13384 consensus_peers.cc:485] T abd272455c064ba48726d933d53c83ed P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.194:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.374470 13384 consensus_peers.cc:485] T caa4af5f41674aaa8587158f25e495fb P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.194:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.383950 13383 consensus_peers.cc:485] T 6395ce4c30eb4644b0a711de3e60b9c0 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.392899 13384 consensus_peers.cc:485] T e62e66aad6d24367a4245e84359a490b P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.393558 13384 consensus_peers.cc:485] T d7db78325b44450a87520a14a2444981 P 88b77f6edc1c4a72b548bf47c05704f3 -> Peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Couldn't send request to peer cd67493f6f0142f896010f2b8a39887d. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.194:33193: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:28.399120 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 13367
W20240502 14:15:28.413398 13515 consensus_peers.cc:485] T 524a9a50c4e843a1ad20f071c83b5aec P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Couldn't send request to peer c86f7440570f4796babfd44c3d7fd847. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.448230 15955 connection.cc:535] client connection to 127.6.196.195:46303 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240502 14:15:28.448664 13514 connection.cc:535] client connection to 127.6.196.195:46303 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:15:28.449138 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 13498
W20240502 14:15:28.449136 15955 consensus_peers.cc:485] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.195:46303: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.449492 13514 consensus_peers.cc:485] T ed5f3b784fa44b09974bc9caf17049af P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.195:46303: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.449569 13514 consensus_peers.cc:485] T 861f9171cb654ec889ddf824f2bb6e92 P 7eb4ecb2379b44c0954f2658c4a04350 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.195:46303: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.453086 15955 consensus_peers.cc:485] T 2363a23cdde54893ae807ef3960dd032 P d1da7f11f0604f78934354a992bb0f55 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.195:46303: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:28.454717 13621 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:28.517096 22087 raft_consensus.cc:482] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:28.517294 22087 raft_consensus.cc:504] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } }
I20240502 14:15:28.517966 22087 leader_election.cc:288] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
W20240502 14:15:28.519207 15957 leader_election.cc:334] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:28.519750 15956 leader_election.cc:334] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:28.519850 15956 leader_election.cc:302] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:28.520074 22087 raft_consensus.cc:2726] T f64bbd1ae9074071825459e558f03ff5 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:28.525756 15957 consensus_peers.cc:485] T eb472259eca84c31b2ff9f33f7e61177 P d1da7f11f0604f78934354a992bb0f55 -> Peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Couldn't send request to peer 5297e816f782430d9a01220f0471a882. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240502 14:15:28.557705 15955 consensus_peers.cc:485] T 4b7d267297334093aa0f65b121b16b61 P d1da7f11f0604f78934354a992bb0f55 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.195:46303: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:28.600764 22087 raft_consensus.cc:482] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 20a967bad5504b1e9c6fb720ecdd1625)
I20240502 14:15:28.600935 22087 raft_consensus.cc:504] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 36687 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } } peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } attrs { promote: false } }
I20240502 14:15:28.601567 22087 leader_election.cc:288] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687), 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857)
W20240502 14:15:28.602699 15955 leader_election.cc:334] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 20a967bad5504b1e9c6fb720ecdd1625 (127.6.196.193:36687): Network error: Client connection negotiation failed: client connection to 127.6.196.193:36687: connect: Connection refused (error 111)
W20240502 14:15:28.602829 15956 leader_election.cc:334] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:28.602945 15956 leader_election.cc:302] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 20a967bad5504b1e9c6fb720ecdd1625, 8b47d4390f8343a493d6c00123785968
I20240502 14:15:28.603149 22087 raft_consensus.cc:2726] T b2905b7940fb45c49ef8b18943dae256 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240502 14:15:28.612838 15955 consensus_peers.cc:485] T df55a71945d34af0affa6eccbdf84dae P d1da7f11f0604f78934354a992bb0f55 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.195:46303: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.639485 15955 consensus_peers.cc:485] T fae5b25ec9694cea87381d27f4d46384 P d1da7f11f0604f78934354a992bb0f55 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.195:46303: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:28.661577 22087 raft_consensus.cc:482] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:28.661737 22087 raft_consensus.cc:504] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } attrs { promote: false } }
I20240502 14:15:28.662305 22087 leader_election.cc:288] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
W20240502 14:15:28.663578 15956 leader_election.cc:334] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
W20240502 14:15:28.663745 15956 leader_election.cc:334] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:28.663815 15956 leader_election.cc:302] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 8b47d4390f8343a493d6c00123785968, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:28.664157 22087 raft_consensus.cc:2726] T 76c85c912ab74c32bfed536ea38d23f6 P d1da7f11f0604f78934354a992bb0f55 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240502 14:15:28.704456 15955 consensus_peers.cc:485] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 -> Peer 88b77f6edc1c4a72b548bf47c05704f3 (127.6.196.195:46303): Couldn't send request to peer 88b77f6edc1c4a72b548bf47c05704f3. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.195:46303: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240502 14:15:28.704456 22087 raft_consensus.cc:482] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c86f7440570f4796babfd44c3d7fd847)
I20240502 14:15:28.704624 22087 raft_consensus.cc:504] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } }
I20240502 14:15:28.705104 22087 leader_election.cc:288] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857), c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041)
W20240502 14:15:28.706128 15956 leader_election.cc:334] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
W20240502 14:15:28.706274 15957 leader_election.cc:334] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
I20240502 14:15:28.706349 15957 leader_election.cc:302] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:28.706557 22087 raft_consensus.cc:2726] T 4ea4a9b24178462dbe84e3c33f2d2245 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:28.720532 22087 raft_consensus.cc:482] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:28.720656 22087 raft_consensus.cc:504] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } } peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } }
I20240502 14:15:28.721201 22087 leader_election.cc:288] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
W20240502 14:15:28.722301 15957 leader_election.cc:334] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857): Network error: Client connection negotiation failed: client connection to 127.6.196.199:33857: connect: Connection refused (error 111)
W20240502 14:15:28.722450 15956 leader_election.cc:334] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041): Network error: Client connection negotiation failed: client connection to 127.6.196.198:42041: connect: Connection refused (error 111)
I20240502 14:15:28.722548 15956 leader_election.cc:302] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 5297e816f782430d9a01220f0471a882, c86f7440570f4796babfd44c3d7fd847
I20240502 14:15:28.722748 22087 raft_consensus.cc:2726] T f021ad6d95174c16a732234f0aa5ad76 P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:28.734041 22087 raft_consensus.cc:482] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cd67493f6f0142f896010f2b8a39887d)
I20240502 14:15:28.734169 22087 raft_consensus.cc:504] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d1da7f11f0604f78934354a992bb0f55" member_type: VOTER last_known_addr { host: "127.6.196.197" port: 43695 } } peers { permanent_uuid: "cd67493f6f0142f896010f2b8a39887d" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 33193 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } }
I20240502 14:15:28.734751 22087 leader_election.cc:288] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193), 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817)
W20240502 14:15:28.735666 15957 leader_election.cc:334] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cd67493f6f0142f896010f2b8a39887d (127.6.196.194:33193): Network error: Client connection negotiation failed: client connection to 127.6.196.194:33193: connect: Connection refused (error 111)
W20240502 14:15:28.757381 15956 consensus_peers.cc:462] T 8a98395ed74c4c2eb265a141aec14bd3 P d1da7f11f0604f78934354a992bb0f55 -> Peer 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.6.196.200:43857: connect: Connection refused (error 111)
I20240502 14:15:28.758785 21255 raft_consensus.cc:482] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8b47d4390f8343a493d6c00123785968)
I20240502 14:15:28.758963 21255 raft_consensus.cc:504] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8b47d4390f8343a493d6c00123785968" member_type: VOTER last_known_addr { host: "127.6.196.200" port: 43857 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } attrs { promote: false } }
I20240502 14:15:28.759737 21255 leader_election.cc:288] T c28e61290d744d4c9409947f9e41706e P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8b47d4390f8343a493d6c00123785968 (127.6.196.200:43857), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:28.760114 21255 raft_consensus.cc:482] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5297e816f782430d9a01220f0471a882)
I20240502 14:15:28.760265 21255 raft_consensus.cc:504] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c86f7440570f4796babfd44c3d7fd847" member_type: VOTER last_known_addr { host: "127.6.196.198" port: 42041 } attrs { replace: true } } peers { permanent_uuid: "7eb4ecb2379b44c0954f2658c4a04350" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 46817 } } peers { permanent_uuid: "5297e816f782430d9a01220f0471a882" member_type: VOTER last_known_addr { host: "127.6.196.199" port: 33857 } } peers { permanent_uuid: "20a967bad5504b1e9c6fb720ecdd1625" member_type: NON_VOTER last_known_addr { host: "127.6.196.193" port: 36687 } attrs { promote: true } }
I20240502 14:15:28.760932 21255 leader_election.cc:288] T 61c065054cdb456da06172650abf0fae P 7eb4ecb2379b44c0954f2658c4a04350 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c86f7440570f4796babfd44c3d7fd847 (127.6.196.198:42041), 5297e816f782430d9a01220f0471a882 (127.6.196.199:33857)
I20240502 14:15:28.767474 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 13498
W20240502 14:15:28.812027 15955 connection.cc:535] client connection to 127.6.196.196:46817 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240502 14:15:28.812786 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 15940
W20240502 14:15:28.813371 15955 consensus_peers.cc:485] T 7a740ff5e0964efcb1fd7dd54dc4094c P d1da7f11f0604f78934354a992bb0f55 -> Peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Couldn't send request to peer 7eb4ecb2379b44c0954f2658c4a04350. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.196:46817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.813495 15955 consensus_peers.cc:485] T 94f535ca99d54963a3fcd54f905281e9 P d1da7f11f0604f78934354a992bb0f55 -> Peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Couldn't send request to peer 7eb4ecb2379b44c0954f2658c4a04350. Status: Network error: Client connection negotiation failed: client connection to 127.6.196.196:46817: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240502 14:15:28.813568 15955 leader_election.cc:334] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 7eb4ecb2379b44c0954f2658c4a04350 (127.6.196.196:46817): Network error: Client connection negotiation failed: client connection to 127.6.196.196:46817: connect: Connection refused (error 111)
I20240502 14:15:28.813653 15955 leader_election.cc:302] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d1da7f11f0604f78934354a992bb0f55; no voters: 7eb4ecb2379b44c0954f2658c4a04350, cd67493f6f0142f896010f2b8a39887d
I20240502 14:15:28.814029 22087 raft_consensus.cc:2726] T 244b205ab48f468a91bf54660ef4711a P d1da7f11f0604f78934354a992bb0f55 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240502 14:15:28.818698 16062 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:29.087992 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 15940
I20240502 14:15:29.132252 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 13014
I20240502 14:15:29.137579 13076 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:29.415750 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 13014
2024-05-02T14:15:29Z chronyd exiting
I20240502 14:15:29.475322 6931 test_util.cc:175] -----------------------------------------------
I20240502 14:15:29.475400 6931 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714659208954437-6931-0
[ FAILED ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1, where GetParam() = 4-byte object <01-00 00-00> (77523 ms)
[----------] 1 test from DDLDuringRebalancingTest (77523 ms total)
[----------] 1 test from RebalancingDuringElectionStormTest
[ RUN ] RebalancingDuringElectionStormTest.RoundRobin/1
I20240502 14:15:29.477329 6931 test_util.cc:268] Using random seed: 1010731696
I20240502 14:15:29.479007 6931 ts_itest-base.cc:112] Starting cluster with:
I20240502 14:15:29.479068 6931 ts_itest-base.cc:113] --------------
I20240502 14:15:29.479116 6931 ts_itest-base.cc:114] 8 tablet servers
I20240502 14:15:29.479175 6931 ts_itest-base.cc:115] 3 replicas per TS
I20240502 14:15:29.479221 6931 ts_itest-base.cc:116] --------------
2024-05-02T14:15:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-02T14:15:29Z Disabled control of system clock
I20240502 14:15:29.510998 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.196.254:33767
--webserver_interface=127.6.196.254
--webserver_port=0
--builtin_ntp_servers=127.6.196.212:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.196.254:33767
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240502 14:15:29.808276 22140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:29.808523 22140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:29.808593 22140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:29.815182 22140 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:29.815281 22140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240502 14:15:29.815325 22140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:29.815368 22140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240502 14:15:29.815410 22140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240502 14:15:29.823570 22140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.6.196.254:33767
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.196.254:33767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.6.196.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:29.824638 22140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:29.826193 22140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:29.836057 22145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:29.836678 22148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:29.837499 22146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:29.838351 22140 server_base.cc:1034] running on GCE node
I20240502 14:15:29.838975 22140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:29.840114 22140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:29.841296 22140 hybrid_clock.cc:648] HybridClock initialized: now 1714659329841257 us; error 36 us; skew 500 ppm
I20240502 14:15:29.841593 22140 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:29.848493 22140 webserver.cc:460] Webserver started at http://127.6.196.254:43919/ using document root <none> and password file <none>
I20240502 14:15:29.849118 22140 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:29.849243 22140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:29.849530 22140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:29.851759 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "7696a0afebd04768bae485398615995d"
format_stamp: "Formatted at 2024-05-02 14:15:29 on dist-test-slave-kjf7"
I20240502 14:15:29.852502 22140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "7696a0afebd04768bae485398615995d"
format_stamp: "Formatted at 2024-05-02 14:15:29 on dist-test-slave-kjf7"
I20240502 14:15:29.856835 22140 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240502 14:15:29.859686 22154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:29.860806 22140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:29.860965 22140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
uuid: "7696a0afebd04768bae485398615995d"
format_stamp: "Formatted at 2024-05-02 14:15:29 on dist-test-slave-kjf7"
I20240502 14:15:29.861145 22140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:29.876389 22140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:29.877024 22140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:29.877458 22140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:29.893625 22140 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.254:33767
I20240502 14:15:29.893632 22205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.254:33767 every 8 connection(s)
I20240502 14:15:29.894701 22140 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240502 14:15:29.897879 22206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:29.898198 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 22140
I20240502 14:15:29.898424 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240502 14:15:29.909564 22206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d: Bootstrap starting.
I20240502 14:15:29.912653 22206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:29.913638 22206 log.cc:824] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:29.915967 22206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d: No bootstrap required, opened a new log
I20240502 14:15:29.921566 22206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7696a0afebd04768bae485398615995d" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 33767 } }
I20240502 14:15:29.921838 22206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:29.921964 22206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7696a0afebd04768bae485398615995d, State: Initialized, Role: FOLLOWER
I20240502 14:15:29.922538 22206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7696a0afebd04768bae485398615995d" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 33767 } }
I20240502 14:15:29.922727 22206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240502 14:15:29.922865 22206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240502 14:15:29.923007 22206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:29.924842 22206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7696a0afebd04768bae485398615995d" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 33767 } }
I20240502 14:15:29.925298 22206 leader_election.cc:302] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7696a0afebd04768bae485398615995d; no voters:
I20240502 14:15:29.925773 22206 leader_election.cc:288] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240502 14:15:29.925889 22211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:29.926481 22211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [term 1 LEADER]: Becoming Leader. State: Replica: 7696a0afebd04768bae485398615995d, State: Running, Role: LEADER
I20240502 14:15:29.926930 22211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7696a0afebd04768bae485398615995d" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 33767 } }
I20240502 14:15:29.927459 22206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [sys.catalog]: configured and running, proceeding with master startup.
I20240502 14:15:29.929556 22212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7696a0afebd04768bae485398615995d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7696a0afebd04768bae485398615995d" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 33767 } } }
I20240502 14:15:29.929481 22213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7696a0afebd04768bae485398615995d. Latest consensus state: current_term: 1 leader_uuid: "7696a0afebd04768bae485398615995d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7696a0afebd04768bae485398615995d" member_type: VOTER last_known_addr { host: "127.6.196.254" port: 33767 } } }
I20240502 14:15:29.929848 22212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [sys.catalog]: This master's current role is: LEADER
I20240502 14:15:29.929859 22213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d [sys.catalog]: This master's current role is: LEADER
I20240502 14:15:29.930804 22218 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240502 14:15:29.936574 22218 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240502 14:15:29.944260 22218 catalog_manager.cc:1347] Generated new cluster ID: 9f934005b2364d449a78fd7106239500
I20240502 14:15:29.944341 22218 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240502 14:15:29.968137 22218 catalog_manager.cc:1370] Generated new certificate authority record
I20240502 14:15:29.969583 22218 catalog_manager.cc:1504] Loading token signing keys...
I20240502 14:15:29.983021 22218 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 7696a0afebd04768bae485398615995d: Generated new TSK 0
I20240502 14:15:29.983879 22218 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240502 14:15:29.991482 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.193:0
--local_ip_for_outbound_sockets=127.6.196.193
--webserver_interface=127.6.196.193
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--builtin_ntp_servers=127.6.196.212:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:15:30.291209 22230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:30.291564 22230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:30.291735 22230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:30.299669 22230 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:30.299794 22230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:30.299958 22230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.193
I20240502 14:15:30.307966 22230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.6.196.193
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.193
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:30.309212 22230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:30.310901 22230 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:30.323004 22235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:30.323580 22236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:30.324386 22238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:30.325112 22230 server_base.cc:1034] running on GCE node
I20240502 14:15:30.325662 22230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:30.326601 22230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:30.327821 22230 hybrid_clock.cc:648] HybridClock initialized: now 1714659330327779 us; error 41 us; skew 500 ppm
I20240502 14:15:30.328138 22230 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:30.330257 22230 webserver.cc:460] Webserver started at http://127.6.196.193:36043/ using document root <none> and password file <none>
I20240502 14:15:30.330929 22230 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:30.331038 22230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:30.331331 22230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:30.333890 22230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2b54c913a94f4c399ec08cf222a941e6"
format_stamp: "Formatted at 2024-05-02 14:15:30 on dist-test-slave-kjf7"
I20240502 14:15:30.334549 22230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2b54c913a94f4c399ec08cf222a941e6"
format_stamp: "Formatted at 2024-05-02 14:15:30 on dist-test-slave-kjf7"
I20240502 14:15:30.339006 22230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240502 14:15:30.341984 22244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:30.343173 22230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20240502 14:15:30.343305 22230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2b54c913a94f4c399ec08cf222a941e6"
format_stamp: "Formatted at 2024-05-02 14:15:30 on dist-test-slave-kjf7"
I20240502 14:15:30.343482 22230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:30.358371 22230 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:30.359048 22230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:30.359277 22230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:30.360073 22230 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:30.361430 22230 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:30.361507 22230 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:30.361584 22230 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:30.361645 22230 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:30.387890 22230 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.193:41463
I20240502 14:15:30.387972 22358 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.193:41463 every 8 connection(s)
I20240502 14:15:30.389035 22230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240502 14:15:30.390592 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 22230
I20240502 14:15:30.390786 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240502 14:15:30.394076 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.194:0
--local_ip_for_outbound_sockets=127.6.196.194
--webserver_interface=127.6.196.194
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--builtin_ntp_servers=127.6.196.212:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:30.400857 22359 heartbeater.cc:347] Connected to a master server at 127.6.196.254:33767
I20240502 14:15:30.401109 22359 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:30.401916 22359 heartbeater.cc:511] Master 127.6.196.254:33767 requested a full tablet report, sending...
I20240502 14:15:30.403951 22171 ts_manager.cc:194] Registered new tserver with Master: 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:30.405880 22171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.193:39527
W20240502 14:15:30.691103 22362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:30.691340 22362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:30.691419 22362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:30.697846 22362 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:30.697943 22362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:30.698040 22362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.194
I20240502 14:15:30.705662 22362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.6.196.194
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.194
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:30.706693 22362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:30.708160 22362 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:30.718654 22367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:30.719400 22368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:30.719934 22370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:30.720650 22362 server_base.cc:1034] running on GCE node
I20240502 14:15:30.721172 22362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:30.722016 22362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:30.723222 22362 hybrid_clock.cc:648] HybridClock initialized: now 1714659330723166 us; error 36 us; skew 500 ppm
I20240502 14:15:30.723482 22362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:30.725402 22362 webserver.cc:460] Webserver started at http://127.6.196.194:38297/ using document root <none> and password file <none>
I20240502 14:15:30.726020 22362 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:30.726123 22362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:30.726409 22362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:30.728658 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "017239f33db84a7b98c8c0244d55ecc5"
format_stamp: "Formatted at 2024-05-02 14:15:30 on dist-test-slave-kjf7"
I20240502 14:15:30.729311 22362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "017239f33db84a7b98c8c0244d55ecc5"
format_stamp: "Formatted at 2024-05-02 14:15:30 on dist-test-slave-kjf7"
I20240502 14:15:30.733631 22362 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20240502 14:15:30.736349 22376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:30.737445 22362 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:30.737578 22362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "017239f33db84a7b98c8c0244d55ecc5"
format_stamp: "Formatted at 2024-05-02 14:15:30 on dist-test-slave-kjf7"
I20240502 14:15:30.737757 22362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:30.752053 22362 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:30.752743 22362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:30.752990 22362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:30.753826 22362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:30.755041 22362 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:30.755124 22362 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:30.755235 22362 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:30.755299 22362 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:30.781773 22362 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.194:39183
I20240502 14:15:30.781814 22489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.194:39183 every 8 connection(s)
I20240502 14:15:30.782959 22362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240502 14:15:30.786109 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 22362
I20240502 14:15:30.786295 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240502 14:15:30.789105 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.195:0
--local_ip_for_outbound_sockets=127.6.196.195
--webserver_interface=127.6.196.195
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--builtin_ntp_servers=127.6.196.212:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:30.794235 22490 heartbeater.cc:347] Connected to a master server at 127.6.196.254:33767
I20240502 14:15:30.794471 22490 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:30.795320 22490 heartbeater.cc:511] Master 127.6.196.254:33767 requested a full tablet report, sending...
I20240502 14:15:30.796650 22171 ts_manager.cc:194] Registered new tserver with Master: 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:30.797344 22171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.194:56935
W20240502 14:15:31.085983 22493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:31.086241 22493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:31.086367 22493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:31.093729 22493 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:31.093878 22493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:31.094091 22493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.195
I20240502 14:15:31.105835 22493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.6.196.195
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.195
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:31.107043 22493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:31.108712 22493 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:31.120563 22501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:31.121459 22499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:31.121898 22498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:31.122567 22493 server_base.cc:1034] running on GCE node
I20240502 14:15:31.123132 22493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:31.124008 22493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:31.125209 22493 hybrid_clock.cc:648] HybridClock initialized: now 1714659331125162 us; error 38 us; skew 500 ppm
I20240502 14:15:31.125520 22493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:31.127622 22493 webserver.cc:460] Webserver started at http://127.6.196.195:35969/ using document root <none> and password file <none>
I20240502 14:15:31.128325 22493 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:31.128436 22493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:31.128768 22493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:31.131219 22493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "00c39ff1d7004a8f8a8847aad597f242"
format_stamp: "Formatted at 2024-05-02 14:15:31 on dist-test-slave-kjf7"
I20240502 14:15:31.132071 22493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "00c39ff1d7004a8f8a8847aad597f242"
format_stamp: "Formatted at 2024-05-02 14:15:31 on dist-test-slave-kjf7"
I20240502 14:15:31.136951 22493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20240502 14:15:31.140201 22508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:31.141479 22493 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:31.141633 22493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "00c39ff1d7004a8f8a8847aad597f242"
format_stamp: "Formatted at 2024-05-02 14:15:31 on dist-test-slave-kjf7"
I20240502 14:15:31.141808 22493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:31.155828 22493 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:31.156493 22493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:31.156721 22493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:31.157536 22493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:31.158767 22493 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:31.158846 22493 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:31.158946 22493 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:31.159011 22493 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:31.187899 22493 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.195:42203
I20240502 14:15:31.187948 22621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.195:42203 every 8 connection(s)
I20240502 14:15:31.188966 22493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240502 14:15:31.198408 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 22493
I20240502 14:15:31.198576 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240502 14:15:31.199268 22622 heartbeater.cc:347] Connected to a master server at 127.6.196.254:33767
I20240502 14:15:31.199491 22622 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:31.200235 22622 heartbeater.cc:511] Master 127.6.196.254:33767 requested a full tablet report, sending...
I20240502 14:15:31.201422 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.196:0
--local_ip_for_outbound_sockets=127.6.196.196
--webserver_interface=127.6.196.196
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--builtin_ntp_servers=127.6.196.212:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:31.201550 22171 ts_manager.cc:194] Registered new tserver with Master: 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:31.202306 22171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.195:46161
I20240502 14:15:31.408387 22359 heartbeater.cc:503] Master 127.6.196.254:33767 was elected leader, sending a full tablet report...
W20240502 14:15:31.498136 22625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:31.498422 22625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:31.498543 22625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:31.505151 22625 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:31.505256 22625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:31.505354 22625 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.196
I20240502 14:15:31.512655 22625 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.6.196.196
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.196
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:31.513739 22625 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:31.515151 22625 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:31.525794 22630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:31.526340 22633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:31.526887 22631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:31.527760 22625 server_base.cc:1034] running on GCE node
I20240502 14:15:31.528268 22625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:31.529196 22625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:31.530423 22625 hybrid_clock.cc:648] HybridClock initialized: now 1714659331530387 us; error 31 us; skew 500 ppm
I20240502 14:15:31.530669 22625 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:31.532732 22625 webserver.cc:460] Webserver started at http://127.6.196.196:44171/ using document root <none> and password file <none>
I20240502 14:15:31.533447 22625 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:31.533535 22625 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:31.533782 22625 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:31.536007 22625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c8a8e31da9734e118e5095919ae17f10"
format_stamp: "Formatted at 2024-05-02 14:15:31 on dist-test-slave-kjf7"
I20240502 14:15:31.536662 22625 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c8a8e31da9734e118e5095919ae17f10"
format_stamp: "Formatted at 2024-05-02 14:15:31 on dist-test-slave-kjf7"
I20240502 14:15:31.541561 22625 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240502 14:15:31.544929 22639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:31.546078 22625 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:31.546207 22625 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c8a8e31da9734e118e5095919ae17f10"
format_stamp: "Formatted at 2024-05-02 14:15:31 on dist-test-slave-kjf7"
I20240502 14:15:31.546366 22625 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:31.559000 22625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:31.559641 22625 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:31.559850 22625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:31.560745 22625 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:31.562110 22625 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:31.562196 22625 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:31.562305 22625 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:31.562376 22625 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:31.592244 22625 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.196:37959
I20240502 14:15:31.592307 22752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.196:37959 every 8 connection(s)
I20240502 14:15:31.593488 22625 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240502 14:15:31.600080 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 22625
I20240502 14:15:31.600265 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240502 14:15:31.604319 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.197:0
--local_ip_for_outbound_sockets=127.6.196.197
--webserver_interface=127.6.196.197
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--builtin_ntp_servers=127.6.196.212:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:31.605720 22753 heartbeater.cc:347] Connected to a master server at 127.6.196.254:33767
I20240502 14:15:31.605959 22753 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:31.606614 22753 heartbeater.cc:511] Master 127.6.196.254:33767 requested a full tablet report, sending...
I20240502 14:15:31.607821 22171 ts_manager.cc:194] Registered new tserver with Master: c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:31.608852 22171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.196:57803
I20240502 14:15:31.799425 22490 heartbeater.cc:503] Master 127.6.196.254:33767 was elected leader, sending a full tablet report...
W20240502 14:15:31.895078 22756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:31.895334 22756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:31.895449 22756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:31.901809 22756 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:31.901935 22756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:31.902078 22756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.197
I20240502 14:15:31.909808 22756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.196.197
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.197
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:31.910856 22756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:31.912328 22756 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:31.922902 22761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:31.923470 22762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:31.923991 22764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:31.924805 22756 server_base.cc:1034] running on GCE node
I20240502 14:15:31.925304 22756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:31.926015 22756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:31.927192 22756 hybrid_clock.cc:648] HybridClock initialized: now 1714659331927157 us; error 25 us; skew 500 ppm
I20240502 14:15:31.927471 22756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:31.929361 22756 webserver.cc:460] Webserver started at http://127.6.196.197:35295/ using document root <none> and password file <none>
I20240502 14:15:31.929940 22756 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:31.930037 22756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:31.930306 22756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:31.932466 22756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "06f023ab0eed430088cc1ce6a17d8418"
format_stamp: "Formatted at 2024-05-02 14:15:31 on dist-test-slave-kjf7"
I20240502 14:15:31.933202 22756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "06f023ab0eed430088cc1ce6a17d8418"
format_stamp: "Formatted at 2024-05-02 14:15:31 on dist-test-slave-kjf7"
I20240502 14:15:31.937547 22756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240502 14:15:31.940224 22770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:31.941435 22756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:31.941548 22756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "06f023ab0eed430088cc1ce6a17d8418"
format_stamp: "Formatted at 2024-05-02 14:15:31 on dist-test-slave-kjf7"
I20240502 14:15:31.941684 22756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:31.955920 22756 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:31.956501 22756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:31.956712 22756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:31.957449 22756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:31.958618 22756 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:31.958690 22756 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:31.958755 22756 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:31.958801 22756 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:31.983803 22756 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.197:42099
I20240502 14:15:31.983867 22884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.197:42099 every 8 connection(s)
I20240502 14:15:31.984874 22756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:15:31.993602 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 22756
I20240502 14:15:31.993753 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240502 14:15:31.995223 22885 heartbeater.cc:347] Connected to a master server at 127.6.196.254:33767
I20240502 14:15:31.995461 22885 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:31.996245 22885 heartbeater.cc:511] Master 127.6.196.254:33767 requested a full tablet report, sending...
I20240502 14:15:31.996811 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.198:0
--local_ip_for_outbound_sockets=127.6.196.198
--webserver_interface=127.6.196.198
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--builtin_ntp_servers=127.6.196.212:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:31.997558 22171 ts_manager.cc:194] Registered new tserver with Master: 06f023ab0eed430088cc1ce6a17d8418 (127.6.196.197:42099)
I20240502 14:15:31.998204 22171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.197:53541
I20240502 14:15:32.204334 22622 heartbeater.cc:503] Master 127.6.196.254:33767 was elected leader, sending a full tablet report...
W20240502 14:15:32.293536 22888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:32.293777 22888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:32.293903 22888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:32.300982 22888 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:32.301116 22888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:32.301327 22888 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.198
I20240502 14:15:32.309727 22888 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.198:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.196.198
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.198
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:32.310791 22888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:32.312345 22888 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:32.323169 22893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:32.323904 22896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:32.324424 22894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:32.325335 22888 server_base.cc:1034] running on GCE node
I20240502 14:15:32.325904 22888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:32.326738 22888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:32.327919 22888 hybrid_clock.cc:648] HybridClock initialized: now 1714659332327868 us; error 56 us; skew 500 ppm
I20240502 14:15:32.328223 22888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:32.330300 22888 webserver.cc:460] Webserver started at http://127.6.196.198:33623/ using document root <none> and password file <none>
I20240502 14:15:32.330988 22888 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:32.331092 22888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:32.331400 22888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:32.334003 22888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "d50d05f10b0c49528cfacfefe923edf2"
format_stamp: "Formatted at 2024-05-02 14:15:32 on dist-test-slave-kjf7"
I20240502 14:15:32.334657 22888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "d50d05f10b0c49528cfacfefe923edf2"
format_stamp: "Formatted at 2024-05-02 14:15:32 on dist-test-slave-kjf7"
I20240502 14:15:32.339200 22888 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20240502 14:15:32.342288 22902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:32.343444 22888 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240502 14:15:32.343578 22888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d50d05f10b0c49528cfacfefe923edf2"
format_stamp: "Formatted at 2024-05-02 14:15:32 on dist-test-slave-kjf7"
I20240502 14:15:32.343739 22888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:32.360769 22888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:32.361501 22888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:32.361706 22888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:32.362519 22888 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:32.363782 22888 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:32.363864 22888 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:32.363955 22888 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:32.364018 22888 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:32.391206 22888 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.198:37151
I20240502 14:15:32.391256 23015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.198:37151 every 8 connection(s)
I20240502 14:15:32.392319 22888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:15:32.394795 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 22888
I20240502 14:15:32.394956 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240502 14:15:32.397783 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.199:0
--local_ip_for_outbound_sockets=127.6.196.199
--webserver_interface=127.6.196.199
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--builtin_ntp_servers=127.6.196.212:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:32.403609 23016 heartbeater.cc:347] Connected to a master server at 127.6.196.254:33767
I20240502 14:15:32.403870 23016 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:32.404608 23016 heartbeater.cc:511] Master 127.6.196.254:33767 requested a full tablet report, sending...
I20240502 14:15:32.405818 22171 ts_manager.cc:194] Registered new tserver with Master: d50d05f10b0c49528cfacfefe923edf2 (127.6.196.198:37151)
I20240502 14:15:32.406505 22171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.198:45737
I20240502 14:15:32.611023 22753 heartbeater.cc:503] Master 127.6.196.254:33767 was elected leader, sending a full tablet report...
W20240502 14:15:32.688642 23019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:32.688861 23019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:32.688935 23019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:32.695133 23019 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:32.695214 23019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:32.695300 23019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.199
I20240502 14:15:32.702710 23019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.199:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.196.199
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.199
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:32.703692 23019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:32.705086 23019 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:32.715683 23025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:32.716481 23027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:32.717272 23024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:32.717885 23019 server_base.cc:1034] running on GCE node
I20240502 14:15:32.718339 23019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:32.719102 23019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:32.720299 23019 hybrid_clock.cc:648] HybridClock initialized: now 1714659332720258 us; error 35 us; skew 500 ppm
I20240502 14:15:32.720559 23019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:32.722620 23019 webserver.cc:460] Webserver started at http://127.6.196.199:33121/ using document root <none> and password file <none>
I20240502 14:15:32.723268 23019 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:32.723384 23019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:32.723654 23019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:32.725849 23019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "e0248ccaeff74f9ea81e0d6181b652ba"
format_stamp: "Formatted at 2024-05-02 14:15:32 on dist-test-slave-kjf7"
I20240502 14:15:32.726486 23019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "e0248ccaeff74f9ea81e0d6181b652ba"
format_stamp: "Formatted at 2024-05-02 14:15:32 on dist-test-slave-kjf7"
I20240502 14:15:32.730738 23019 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240502 14:15:32.733629 23033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:32.734653 23019 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20240502 14:15:32.734774 23019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e0248ccaeff74f9ea81e0d6181b652ba"
format_stamp: "Formatted at 2024-05-02 14:15:32 on dist-test-slave-kjf7"
I20240502 14:15:32.734918 23019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:32.748915 23019 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:32.749612 23019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:32.749836 23019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:32.750649 23019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:32.751818 23019 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:32.751894 23019 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:32.751991 23019 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:32.752053 23019 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:32.780351 23019 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.199:43327
I20240502 14:15:32.780398 23147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.199:43327 every 8 connection(s)
I20240502 14:15:32.781699 23019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:15:32.788395 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 23019
I20240502 14:15:32.788537 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240502 14:15:32.792261 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.200:0
--local_ip_for_outbound_sockets=127.6.196.200
--webserver_interface=127.6.196.200
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--builtin_ntp_servers=127.6.196.212:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:32.793092 23148 heartbeater.cc:347] Connected to a master server at 127.6.196.254:33767
I20240502 14:15:32.793370 23148 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:32.794035 23148 heartbeater.cc:511] Master 127.6.196.254:33767 requested a full tablet report, sending...
I20240502 14:15:32.795219 22171 ts_manager.cc:194] Registered new tserver with Master: e0248ccaeff74f9ea81e0d6181b652ba (127.6.196.199:43327)
I20240502 14:15:32.796154 22171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.199:54749
I20240502 14:15:33.000155 22885 heartbeater.cc:503] Master 127.6.196.254:33767 was elected leader, sending a full tablet report...
W20240502 14:15:33.079039 23151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:33.079273 23151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:33.079345 23151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:33.086287 23151 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:33.086381 23151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:33.086481 23151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.200
I20240502 14:15:33.094399 23151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.200:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.196.200
--webserver_port=0
--tserver_master_addrs=127.6.196.254:33767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.200
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:33.095649 23151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:33.097666 23151 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:33.108994 23160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:33.109503 23157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:33.110224 23156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:33.111040 23151 server_base.cc:1034] running on GCE node
I20240502 14:15:33.111553 23151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:33.112385 23151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:33.113554 23151 hybrid_clock.cc:648] HybridClock initialized: now 1714659333113512 us; error 37 us; skew 500 ppm
I20240502 14:15:33.113795 23151 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:33.115674 23151 webserver.cc:460] Webserver started at http://127.6.196.200:39901/ using document root <none> and password file <none>
I20240502 14:15:33.116317 23151 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:33.116411 23151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:33.116691 23151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240502 14:15:33.119068 23151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "7eeefbc07b7c48e09fa850ee106bd0ca"
format_stamp: "Formatted at 2024-05-02 14:15:33 on dist-test-slave-kjf7"
I20240502 14:15:33.119758 23151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "7eeefbc07b7c48e09fa850ee106bd0ca"
format_stamp: "Formatted at 2024-05-02 14:15:33 on dist-test-slave-kjf7"
I20240502 14:15:33.124276 23151 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240502 14:15:33.127022 23166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:33.128155 23151 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:33.128298 23151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "7eeefbc07b7c48e09fa850ee106bd0ca"
format_stamp: "Formatted at 2024-05-02 14:15:33 on dist-test-slave-kjf7"
I20240502 14:15:33.128455 23151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:33.149333 23151 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:33.149955 23151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:33.150178 23151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:33.150934 23151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:33.152180 23151 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:33.152292 23151 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:33.152401 23151 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:33.152473 23151 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:33.178125 23151 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.200:35297
I20240502 14:15:33.178174 23279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.200:35297 every 8 connection(s)
I20240502 14:15:33.179128 23151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:15:33.181795 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 23151
I20240502 14:15:33.182003 6931 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240502 14:15:33.189851 23280 heartbeater.cc:347] Connected to a master server at 127.6.196.254:33767
I20240502 14:15:33.190076 23280 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:33.190685 23280 heartbeater.cc:511] Master 127.6.196.254:33767 requested a full tablet report, sending...
I20240502 14:15:33.191764 22171 ts_manager.cc:194] Registered new tserver with Master: 7eeefbc07b7c48e09fa850ee106bd0ca (127.6.196.200:35297)
I20240502 14:15:33.192483 22171 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.200:34987
I20240502 14:15:33.197098 6931 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:15:33.210736 6931 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240502 14:15:33.210824 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 22756
I20240502 14:15:33.217550 22879 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:33.290776 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 22756
I20240502 14:15:33.308281 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 22888
I20240502 14:15:33.315362 23011 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:33.392922 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 22888
I20240502 14:15:33.408663 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 23019
I20240502 14:15:33.415796 23143 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:33.488319 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 23019
I20240502 14:15:33.504477 6931 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu pid 23151
I20240502 14:15:33.511790 23275 generic_service.cc:196] Checking for leaks (request via RPC)
I20240502 14:15:33.586767 6931 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu with pid 23151
I20240502 14:15:37.355424 22171 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:37.402468 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 4a15c7effaee47b992ffd5078bf7ca9d (DEFAULT_TABLE table=rebalance_test_table_0 [id=4af62cc6bbbb4f67907086dcad0179b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:37.404238 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a15c7effaee47b992ffd5078bf7ca9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:37.407320 22555 tablet_service.cc:1467] Processing CreateTablet for tablet 84ccfc433f3f4cc4bbfcba08574f6c3f (DEFAULT_TABLE table=rebalance_test_table_0 [id=4af62cc6bbbb4f67907086dcad0179b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:37.407855 22555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ccfc433f3f4cc4bbfcba08574f6c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:37.408167 22423 tablet_service.cc:1467] Processing CreateTablet for tablet 84ccfc433f3f4cc4bbfcba08574f6c3f (DEFAULT_TABLE table=rebalance_test_table_0 [id=4af62cc6bbbb4f67907086dcad0179b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:37.408381 22422 tablet_service.cc:1467] Processing CreateTablet for tablet 65879e88cc294682a159d1370a11c3f7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4af62cc6bbbb4f67907086dcad0179b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:37.407683 22424 tablet_service.cc:1467] Processing CreateTablet for tablet 4a15c7effaee47b992ffd5078bf7ca9d (DEFAULT_TABLE table=rebalance_test_table_0 [id=4af62cc6bbbb4f67907086dcad0179b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:37.409188 22423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ccfc433f3f4cc4bbfcba08574f6c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:37.409626 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a15c7effaee47b992ffd5078bf7ca9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:37.413374 22422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65879e88cc294682a159d1370a11c3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:37.413136 22291 tablet_service.cc:1467] Processing CreateTablet for tablet 65879e88cc294682a159d1370a11c3f7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4af62cc6bbbb4f67907086dcad0179b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:37.414285 22292 tablet_service.cc:1467] Processing CreateTablet for tablet 84ccfc433f3f4cc4bbfcba08574f6c3f (DEFAULT_TABLE table=rebalance_test_table_0 [id=4af62cc6bbbb4f67907086dcad0179b5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:37.414462 22291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65879e88cc294682a159d1370a11c3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:37.414867 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84ccfc433f3f4cc4bbfcba08574f6c3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:37.431092 22688 tablet_service.cc:1467] Processing CreateTablet for tablet 4a15c7effaee47b992ffd5078bf7ca9d (DEFAULT_TABLE table=rebalance_test_table_0 [id=4af62cc6bbbb4f67907086dcad0179b5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:37.428275 23311 tablet_bootstrap.cc:492] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:37.432490 22686 tablet_service.cc:1467] Processing CreateTablet for tablet 65879e88cc294682a159d1370a11c3f7 (DEFAULT_TABLE table=rebalance_test_table_0 [id=4af62cc6bbbb4f67907086dcad0179b5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:37.432955 22688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a15c7effaee47b992ffd5078bf7ca9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:37.433478 22686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65879e88cc294682a159d1370a11c3f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:37.436367 23311 tablet_bootstrap.cc:654] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:37.437516 23311 log.cc:824] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:37.449206 23313 tablet_bootstrap.cc:492] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:37.453989 23313 tablet_bootstrap.cc:654] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:37.455524 23310 tablet_bootstrap.cc:492] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:37.456037 23313 log.cc:824] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:37.456691 23314 tablet_bootstrap.cc:492] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:37.459856 23310 tablet_bootstrap.cc:654] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:37.460541 23314 tablet_bootstrap.cc:654] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:37.461561 23310 log.cc:824] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:37.462692 23314 log.cc:824] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10: Log is configured to *not* fsync() on all Append() calls
I20240502 14:15:37.464167 23311 tablet_bootstrap.cc:492] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:37.464416 23311 ts_tablet_manager.cc:1367] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.036s user 0.014s sys 0.014s
I20240502 14:15:37.465451 23313 tablet_bootstrap.cc:492] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:37.465700 23313 ts_tablet_manager.cc:1367] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
I20240502 14:15:37.465929 23314 tablet_bootstrap.cc:492] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:37.466174 23314 ts_tablet_manager.cc:1367] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20240502 14:15:37.467211 23310 tablet_bootstrap.cc:492] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:37.467523 23310 ts_tablet_manager.cc:1367] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.009s
I20240502 14:15:37.472493 23311 raft_consensus.cc:348] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:37.472931 23311 raft_consensus.cc:374] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:37.473029 23311 raft_consensus.cc:729] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:37.473774 23311 consensus_queue.cc:260] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:37.473908 23314 raft_consensus.cc:348] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:37.474298 23314 raft_consensus.cc:374] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:37.474177 23313 raft_consensus.cc:348] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:37.474421 23314 raft_consensus.cc:729] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:37.474579 23313 raft_consensus.cc:374] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:37.474686 23313 raft_consensus.cc:729] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:37.475241 23314 consensus_queue.cc:260] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:37.475399 23313 consensus_queue.cc:260] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:37.476636 23310 raft_consensus.cc:348] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:37.477319 23310 raft_consensus.cc:374] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:37.477489 23310 raft_consensus.cc:729] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:37.478360 23310 consensus_queue.cc:260] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:37.480376 23314 ts_tablet_manager.cc:1398] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.014s user 0.004s sys 0.009s
I20240502 14:15:37.483692 23313 ts_tablet_manager.cc:1398] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20240502 14:15:37.487169 23314 tablet_bootstrap.cc:492] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:37.489145 23313 tablet_bootstrap.cc:492] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:37.489817 23314 tablet_bootstrap.cc:654] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:37.491493 23313 tablet_bootstrap.cc:654] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:37.491986 23311 ts_tablet_manager.cc:1398] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.027s user 0.012s sys 0.016s
I20240502 14:15:37.492388 23311 tablet_bootstrap.cc:492] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:37.494299 23311 tablet_bootstrap.cc:654] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:37.497499 23310 ts_tablet_manager.cc:1398] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.030s user 0.017s sys 0.005s
I20240502 14:15:37.498116 23310 tablet_bootstrap.cc:492] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:37.500351 23310 tablet_bootstrap.cc:654] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:37.501626 23313 tablet_bootstrap.cc:492] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:37.501765 23313 ts_tablet_manager.cc:1367] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240502 14:15:37.501838 23314 tablet_bootstrap.cc:492] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:37.501996 23314 ts_tablet_manager.cc:1367] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240502 14:15:37.502503 23310 tablet_bootstrap.cc:492] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:37.502661 23310 ts_tablet_manager.cc:1367] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240502 14:15:37.502930 23311 tablet_bootstrap.cc:492] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:37.502862 23313 raft_consensus.cc:348] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:37.503062 23311 ts_tablet_manager.cc:1367] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20240502 14:15:37.502959 23314 raft_consensus.cc:348] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:37.503078 23313 raft_consensus.cc:374] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:37.503156 23314 raft_consensus.cc:374] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:37.503172 23313 raft_consensus.cc:729] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:37.503250 23314 raft_consensus.cc:729] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:37.503417 23313 consensus_queue.cc:260] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:37.503525 23314 consensus_queue.cc:260] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:37.503662 23310 raft_consensus.cc:348] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:37.503885 23310 raft_consensus.cc:374] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:37.503962 23310 raft_consensus.cc:729] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:37.503996 23313 ts_tablet_manager.cc:1398] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:37.503993 23311 raft_consensus.cc:348] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:37.504164 23311 raft_consensus.cc:374] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:37.504245 23311 raft_consensus.cc:729] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:37.504249 23310 consensus_queue.cc:260] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:37.504431 23311 consensus_queue.cc:260] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:37.504727 23310 ts_tablet_manager.cc:1398] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:37.504863 23311 ts_tablet_manager.cc:1398] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:37.505123 23311 tablet_bootstrap.cc:492] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:37.506523 23311 tablet_bootstrap.cc:654] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:37.506623 23314 ts_tablet_manager.cc:1398] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:37.508491 23311 tablet_bootstrap.cc:492] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:37.508666 23311 ts_tablet_manager.cc:1367] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:15:37.509402 23311 raft_consensus.cc:348] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:37.509639 23311 raft_consensus.cc:374] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:37.509742 23311 raft_consensus.cc:729] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:37.509966 23311 consensus_queue.cc:260] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:37.510418 23311 ts_tablet_manager.cc:1398] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240502 14:15:37.539287 22491 tablet.cc:2369] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:15:37.582563 23321 raft_consensus.cc:482] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:37.582811 23321 raft_consensus.cc:504] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:37.584420 23321 leader_election.cc:288] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:37.592418 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:37.592402 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:37.592962 22708 raft_consensus.cc:2454] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:37.592957 22445 raft_consensus.cc:2454] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:37.593549 22512 leader_election.cc:302] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:37.593930 23321 raft_consensus.cc:2781] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:37.594063 23321 raft_consensus.cc:482] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:37.594156 23321 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:37.595949 23321 raft_consensus.cc:504] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:37.596447 23321 leader_election.cc:288] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:37.596913 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:37.596915 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:37.597082 22708 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:37.597098 22445 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
W20240502 14:15:37.598945 22754 tablet.cc:2369] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:15:37.599449 22708 raft_consensus.cc:2454] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:37.599808 22445 raft_consensus.cc:2454] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:37.599936 22509 leader_election.cc:302] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:37.600289 23321 raft_consensus.cc:2781] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:37.600751 23321 raft_consensus.cc:686] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:37.601253 23321 consensus_queue.cc:237] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:37.605517 22169 catalog_manager.cc:5484] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:37.609650 23321 raft_consensus.cc:482] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:37.609776 23321 raft_consensus.cc:504] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:37.610278 23321 leader_election.cc:288] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:37.610610 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:37.610853 22445 raft_consensus.cc:2454] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:37.611171 22512 leader_election.cc:302] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:37.611451 23321 raft_consensus.cc:2781] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:37.611529 23321 raft_consensus.cc:482] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:37.611624 23321 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:37.613725 23321 raft_consensus.cc:504] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:37.614302 23321 leader_election.cc:288] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:37.614630 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:37.614809 22445 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:37.616266 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:37.616287 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:37.616492 22445 raft_consensus.cc:2454] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:37.616796 22312 raft_consensus.cc:2454] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:37.616878 22512 leader_election.cc:302] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:37.617353 23321 raft_consensus.cc:2781] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:37.617532 23321 raft_consensus.cc:686] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:37.617766 23321 consensus_queue.cc:237] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:37.619791 22169 catalog_manager.cc:5484] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:37.645138 22360 tablet.cc:2369] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240502 14:15:37.694245 22623 tablet.cc:2369] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240502 14:15:37.704883 23320 raft_consensus.cc:482] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:37.705024 23320 raft_consensus.cc:504] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:37.706440 23320 leader_election.cc:288] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:37.712721 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:37.712920 22445 raft_consensus.cc:2454] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 0.
I20240502 14:15:37.712954 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:37.713212 22708 raft_consensus.cc:2454] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 0.
I20240502 14:15:37.713361 22248 leader_election.cc:302] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:37.713748 23320 raft_consensus.cc:2781] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:37.713901 23320 raft_consensus.cc:482] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:37.713991 23320 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:37.715889 23320 raft_consensus.cc:504] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:37.716362 23320 leader_election.cc:288] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:37.716681 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:37.716730 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:37.716846 22445 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:37.716943 22708 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:37.718578 22445 raft_consensus.cc:2454] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 1.
I20240502 14:15:37.718679 22708 raft_consensus.cc:2454] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 1.
I20240502 14:15:37.718906 22248 leader_election.cc:302] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:37.719143 23320 raft_consensus.cc:2781] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:37.719475 23320 raft_consensus.cc:686] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:37.719844 23320 consensus_queue.cc:237] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:37.723655 22169 catalog_manager.cc:5484] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:38.030648 23321 consensus_queue.cc:1035] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:38.039927 23321 consensus_queue.cc:1035] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:15:38.055284 23336 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:38.055539 23336 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:38.062393 23336 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240502 14:15:38.177824 22311 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:38.178002 22445 raft_consensus.cc:1261] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:38.178550 23342 consensus_queue.cc:1035] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:38.182577 22311 raft_consensus.cc:1261] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:38.197072 23342 consensus_queue.cc:1035] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:38.244328 23340 mvcc.cc:205] Tried to move back new op lower bound from 7023244649167720448 to 7023244646884196352. Current Snapshot: MvccSnapshot[applied={T|T < 7023244649167720448}]
I20240502 14:15:38.256353 22708 raft_consensus.cc:1261] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:38.257191 22443 raft_consensus.cc:1261] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:38.258473 23320 consensus_queue.cc:1035] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:38.259204 23320 consensus_queue.cc:1035] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:38.288034 23388 mvcc.cc:205] Tried to move back new op lower bound from 7023244649167720448 to 7023244646884196352. Current Snapshot: MvccSnapshot[applied={T|T < 7023244649167720448}]
I20240502 14:15:38.313524 23379 mvcc.cc:205] Tried to move back new op lower bound from 7023244649167720448 to 7023244646884196352. Current Snapshot: MvccSnapshot[applied={T|T < 7023244649167720448}]
INSERT report
rows total: 2000
time total: 255.208 ms
time per row: 0.127604 ms
I20240502 14:15:38.456781 22169 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:38.465052 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 7951641f78914cc583da7f0c4b99ed9d (DEFAULT_TABLE table=rebalance_test_table_1 [id=0fce405174dd4a2bbfc24029d8065880]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:38.465523 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7951641f78914cc583da7f0c4b99ed9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:38.465456 22291 tablet_service.cc:1467] Processing CreateTablet for tablet 7951641f78914cc583da7f0c4b99ed9d (DEFAULT_TABLE table=rebalance_test_table_1 [id=0fce405174dd4a2bbfc24029d8065880]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:38.465466 22686 tablet_service.cc:1467] Processing CreateTablet for tablet 7951641f78914cc583da7f0c4b99ed9d (DEFAULT_TABLE table=rebalance_test_table_1 [id=0fce405174dd4a2bbfc24029d8065880]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:38.465855 22686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7951641f78914cc583da7f0c4b99ed9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:38.465857 22291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7951641f78914cc583da7f0c4b99ed9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:38.466802 22688 tablet_service.cc:1467] Processing CreateTablet for tablet 44f40d47e54a429a9f55ae410e70ddf3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0fce405174dd4a2bbfc24029d8065880]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:38.467161 22688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f40d47e54a429a9f55ae410e70ddf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:38.470125 22424 tablet_service.cc:1467] Processing CreateTablet for tablet f344dcd458ee4654978c625337ff0bab (DEFAULT_TABLE table=rebalance_test_table_1 [id=0fce405174dd4a2bbfc24029d8065880]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:38.470516 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f344dcd458ee4654978c625337ff0bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:38.474166 22292 tablet_service.cc:1467] Processing CreateTablet for tablet 44f40d47e54a429a9f55ae410e70ddf3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0fce405174dd4a2bbfc24029d8065880]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:38.474628 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f40d47e54a429a9f55ae410e70ddf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:38.477869 22290 tablet_service.cc:1467] Processing CreateTablet for tablet f344dcd458ee4654978c625337ff0bab (DEFAULT_TABLE table=rebalance_test_table_1 [id=0fce405174dd4a2bbfc24029d8065880]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:38.478382 22290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f344dcd458ee4654978c625337ff0bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:38.481979 22555 tablet_service.cc:1467] Processing CreateTablet for tablet 44f40d47e54a429a9f55ae410e70ddf3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0fce405174dd4a2bbfc24029d8065880]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:38.482394 22555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f40d47e54a429a9f55ae410e70ddf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:38.482998 23408 tablet_bootstrap.cc:492] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:38.485189 23408 tablet_bootstrap.cc:654] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:38.485353 23409 tablet_bootstrap.cc:492] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:38.485723 22554 tablet_service.cc:1467] Processing CreateTablet for tablet f344dcd458ee4654978c625337ff0bab (DEFAULT_TABLE table=rebalance_test_table_1 [id=0fce405174dd4a2bbfc24029d8065880]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:38.486181 22554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f344dcd458ee4654978c625337ff0bab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:38.487228 23409 tablet_bootstrap.cc:654] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:38.489310 23407 tablet_bootstrap.cc:492] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:38.491652 23408 tablet_bootstrap.cc:492] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:38.491820 23408 ts_tablet_manager.cc:1367] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240502 14:15:38.492183 23407 tablet_bootstrap.cc:654] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:38.492558 23408 raft_consensus.cc:348] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.492825 23408 raft_consensus.cc:374] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:38.492962 23408 raft_consensus.cc:729] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:38.493248 23408 consensus_queue.cc:260] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.493919 23408 ts_tablet_manager.cc:1398] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:38.501597 23407 tablet_bootstrap.cc:492] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:38.501828 23407 ts_tablet_manager.cc:1367] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240502 14:15:38.502902 23407 raft_consensus.cc:348] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.503252 23407 raft_consensus.cc:374] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:38.503417 23407 raft_consensus.cc:729] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:38.504344 23408 tablet_bootstrap.cc:492] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:38.505168 23407 consensus_queue.cc:260] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.505867 23408 tablet_bootstrap.cc:654] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:38.506536 23407 ts_tablet_manager.cc:1398] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:38.506835 23407 tablet_bootstrap.cc:492] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:38.508733 23407 tablet_bootstrap.cc:654] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:38.514192 23408 tablet_bootstrap.cc:492] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:38.514277 23406 tablet_bootstrap.cc:492] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:38.514345 23408 ts_tablet_manager.cc:1367] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20240502 14:15:38.514562 23407 tablet_bootstrap.cc:492] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:38.514695 23407 ts_tablet_manager.cc:1367] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240502 14:15:38.514917 23408 raft_consensus.cc:348] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.515071 23408 raft_consensus.cc:374] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:38.515149 23408 raft_consensus.cc:729] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:38.515359 23408 consensus_queue.cc:260] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.515342 23407 raft_consensus.cc:348] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.515535 23407 raft_consensus.cc:374] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:38.515601 23407 raft_consensus.cc:729] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:38.515884 23406 tablet_bootstrap.cc:654] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:38.515812 23407 consensus_queue.cc:260] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.516291 23407 ts_tablet_manager.cc:1398] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:38.517364 23408 ts_tablet_manager.cc:1398] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:38.517642 23408 tablet_bootstrap.cc:492] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:38.519105 23406 tablet_bootstrap.cc:492] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:38.519264 23406 ts_tablet_manager.cc:1367] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:15:38.519217 23408 tablet_bootstrap.cc:654] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:38.519754 23409 tablet_bootstrap.cc:492] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:38.519971 23409 ts_tablet_manager.cc:1367] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.035s user 0.004s sys 0.022s
I20240502 14:15:38.520015 23406 raft_consensus.cc:348] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.520206 23406 raft_consensus.cc:374] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:38.520294 23406 raft_consensus.cc:729] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:38.520535 23406 consensus_queue.cc:260] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.521027 23406 ts_tablet_manager.cc:1398] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:38.521271 23408 tablet_bootstrap.cc:492] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:38.521350 23406 tablet_bootstrap.cc:492] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:38.520785 23409 raft_consensus.cc:348] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:38.521562 23408 ts_tablet_manager.cc:1367] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240502 14:15:38.521559 23409 raft_consensus.cc:374] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:38.521649 23409 raft_consensus.cc:729] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:38.521947 23409 consensus_queue.cc:260] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:38.522642 23406 tablet_bootstrap.cc:654] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:38.522810 23408 raft_consensus.cc:348] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:38.523072 23408 raft_consensus.cc:374] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:38.523233 23408 raft_consensus.cc:729] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:38.523540 23408 consensus_queue.cc:260] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:38.524763 23409 ts_tablet_manager.cc:1398] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:15:38.526131 23408 ts_tablet_manager.cc:1398] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:15:38.526528 23406 tablet_bootstrap.cc:492] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:38.526688 23406 ts_tablet_manager.cc:1367] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240502 14:15:38.527570 23406 raft_consensus.cc:348] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:38.527791 23406 raft_consensus.cc:374] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:38.527889 23406 raft_consensus.cc:729] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:38.528128 23406 consensus_queue.cc:260] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:38.528859 23406 ts_tablet_manager.cc:1398] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:38.529187 23406 tablet_bootstrap.cc:492] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:38.530313 23406 tablet_bootstrap.cc:654] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:38.531831 23406 tablet_bootstrap.cc:492] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:38.531925 23406 ts_tablet_manager.cc:1367] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:38.532536 23406 raft_consensus.cc:348] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.532707 23406 raft_consensus.cc:374] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:38.532794 23406 raft_consensus.cc:729] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:38.532971 23406 consensus_queue.cc:260] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.533457 23406 ts_tablet_manager.cc:1398] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240502 14:15:38.547214 23342 raft_consensus.cc:482] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:38.547319 23342 raft_consensus.cc:504] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:38.547840 23342 leader_election.cc:288] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:38.548213 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:38.548255 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:38.548425 22445 raft_consensus.cc:2454] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:38.548421 22312 raft_consensus.cc:2454] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:38.548732 22509 leader_election.cc:302] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:38.548960 23342 raft_consensus.cc:2781] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:38.549036 23342 raft_consensus.cc:482] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:38.549088 23342 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:38.550657 23342 raft_consensus.cc:504] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:38.551080 23342 leader_election.cc:288] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:38.551481 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:38.551501 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:38.551622 22445 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:38.551631 22312 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:38.553471 22445 raft_consensus.cc:2454] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:38.553555 22312 raft_consensus.cc:2454] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:38.553826 22512 leader_election.cc:302] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:38.554203 23342 raft_consensus.cc:2781] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:38.554314 23342 raft_consensus.cc:686] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:38.554553 23342 consensus_queue.cc:237] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:38.556718 22168 catalog_manager.cc:5484] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:38.588321 23343 raft_consensus.cc:482] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:38.588507 23343 raft_consensus.cc:504] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.590042 23343 leader_election.cc:288] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:38.597151 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40d47e54a429a9f55ae410e70ddf3" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:38.597429 22312 raft_consensus.cc:2454] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:38.598100 22640 leader_election.cc:302] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:38.598176 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40d47e54a429a9f55ae410e70ddf3" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242" is_pre_election: true
I20240502 14:15:38.598526 23343 raft_consensus.cc:2781] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:38.598567 22576 raft_consensus.cc:2454] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:38.598626 23343 raft_consensus.cc:482] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:38.598717 23343 raft_consensus.cc:3037] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:38.601006 23343 raft_consensus.cc:504] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.601670 23343 leader_election.cc:288] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:38.602083 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40d47e54a429a9f55ae410e70ddf3" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:38.602162 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40d47e54a429a9f55ae410e70ddf3" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:38.602298 22312 raft_consensus.cc:3037] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:38.602339 22576 raft_consensus.cc:3037] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:38.604923 22312 raft_consensus.cc:2454] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:38.605147 22576 raft_consensus.cc:2454] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:38.605351 22640 leader_election.cc:302] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:38.605607 23343 raft_consensus.cc:2781] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:38.606019 23343 raft_consensus.cc:686] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:38.606426 23343 consensus_queue.cc:237] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.611650 22168 catalog_manager.cc:5484] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:38.758504 23343 raft_consensus.cc:482] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:38.758708 23343 raft_consensus.cc:504] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.759279 23343 leader_election.cc:288] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:38.759732 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7951641f78914cc583da7f0c4b99ed9d" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242" is_pre_election: true
I20240502 14:15:38.759810 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7951641f78914cc583da7f0c4b99ed9d" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:38.759995 22576 raft_consensus.cc:2454] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:38.760058 22312 raft_consensus.cc:2454] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:38.760411 22642 leader_election.cc:302] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:38.760684 23343 raft_consensus.cc:2781] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:38.760774 23343 raft_consensus.cc:482] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:38.760874 23343 raft_consensus.cc:3037] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:38.762560 23343 raft_consensus.cc:504] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.763106 23343 leader_election.cc:288] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:38.763422 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7951641f78914cc583da7f0c4b99ed9d" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:38.763646 22576 raft_consensus.cc:3037] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:38.763619 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7951641f78914cc583da7f0c4b99ed9d" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:38.763818 22312 raft_consensus.cc:3037] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:38.766175 22576 raft_consensus.cc:2454] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:38.766219 22312 raft_consensus.cc:2454] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:38.766585 22640 leader_election.cc:302] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:38.766826 23343 raft_consensus.cc:2781] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:38.766997 23343 raft_consensus.cc:686] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:38.767293 23343 consensus_queue.cc:237] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:38.769713 22168 catalog_manager.cc:5484] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:38.998749 23342 consensus_queue.cc:1035] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:39.001579 23343 consensus_queue.cc:1035] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:39.007858 23329 consensus_queue.cc:1035] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:39.014875 23343 consensus_queue.cc:1035] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:15:39.156510 23426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:39.156769 23426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:39.163807 23426 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240502 14:15:39.282802 22311 raft_consensus.cc:1261] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:39.283587 23423 consensus_queue.cc:1035] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:39.295567 22576 raft_consensus.cc:1261] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:39.296434 23423 consensus_queue.cc:1035] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:39.384084 23462 mvcc.cc:205] Tried to move back new op lower bound from 7023244653694799872 to 7023244651592884224. Current Snapshot: MvccSnapshot[applied={T|T < 7023244653694799872}]
INSERT report
rows total: 2000
time total: 212.009 ms
time per row: 0.106004 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:15:40.379453 22168 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:40.387940 22292 tablet_service.cc:1467] Processing CreateTablet for tablet 3c962be661024d9aae9d8d613763cc49 (DEFAULT_TABLE table=rebalance_test_table_2 [id=277c88b6b7814251988c44781da3dccd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:40.388182 22424 tablet_service.cc:1467] Processing CreateTablet for tablet 3c962be661024d9aae9d8d613763cc49 (DEFAULT_TABLE table=rebalance_test_table_2 [id=277c88b6b7814251988c44781da3dccd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:40.388389 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c962be661024d9aae9d8d613763cc49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:40.388397 22688 tablet_service.cc:1467] Processing CreateTablet for tablet 3c962be661024d9aae9d8d613763cc49 (DEFAULT_TABLE table=rebalance_test_table_2 [id=277c88b6b7814251988c44781da3dccd]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:40.388586 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c962be661024d9aae9d8d613763cc49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:40.388757 22688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c962be661024d9aae9d8d613763cc49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:40.389989 22686 tablet_service.cc:1467] Processing CreateTablet for tablet 0af05ecb918d4f95ae1847dbe62e73fc (DEFAULT_TABLE table=rebalance_test_table_2 [id=277c88b6b7814251988c44781da3dccd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:40.390319 22686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af05ecb918d4f95ae1847dbe62e73fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:40.393674 22685 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea274a5cbef491b9803ace227431a54 (DEFAULT_TABLE table=rebalance_test_table_2 [id=277c88b6b7814251988c44781da3dccd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:40.394052 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea274a5cbef491b9803ace227431a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:40.394320 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 0af05ecb918d4f95ae1847dbe62e73fc (DEFAULT_TABLE table=rebalance_test_table_2 [id=277c88b6b7814251988c44781da3dccd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:40.395094 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af05ecb918d4f95ae1847dbe62e73fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:40.396617 22290 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea274a5cbef491b9803ace227431a54 (DEFAULT_TABLE table=rebalance_test_table_2 [id=277c88b6b7814251988c44781da3dccd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:40.396967 22290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea274a5cbef491b9803ace227431a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:40.400419 23487 tablet_bootstrap.cc:492] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:40.400597 22422 tablet_service.cc:1467] Processing CreateTablet for tablet 0af05ecb918d4f95ae1847dbe62e73fc (DEFAULT_TABLE table=rebalance_test_table_2 [id=277c88b6b7814251988c44781da3dccd]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:40.400972 22422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0af05ecb918d4f95ae1847dbe62e73fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:40.403218 23487 tablet_bootstrap.cc:654] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:40.405548 22423 tablet_service.cc:1467] Processing CreateTablet for tablet 5ea274a5cbef491b9803ace227431a54 (DEFAULT_TABLE table=rebalance_test_table_2 [id=277c88b6b7814251988c44781da3dccd]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:40.405915 22423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ea274a5cbef491b9803ace227431a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:40.413516 23487 tablet_bootstrap.cc:492] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:40.413755 23487 ts_tablet_manager.cc:1367] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.009s
I20240502 14:15:40.414710 23487 raft_consensus.cc:348] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:40.415079 23487 raft_consensus.cc:374] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:40.415230 23487 raft_consensus.cc:729] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:40.415581 23487 consensus_queue.cc:260] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:40.418972 23487 ts_tablet_manager.cc:1398] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:40.421818 23491 tablet_bootstrap.cc:492] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:40.421968 23490 tablet_bootstrap.cc:492] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:40.422354 23487 tablet_bootstrap.cc:492] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:40.423525 23491 tablet_bootstrap.cc:654] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:40.424640 23487 tablet_bootstrap.cc:654] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:40.425516 23488 tablet_bootstrap.cc:492] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:40.426810 23488 tablet_bootstrap.cc:654] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:40.428300 23491 tablet_bootstrap.cc:492] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:40.428443 23491 ts_tablet_manager.cc:1367] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240502 14:15:40.429117 23491 raft_consensus.cc:348] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:40.429307 23490 tablet_bootstrap.cc:654] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:40.429412 23491 raft_consensus.cc:374] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:40.430109 23488 tablet_bootstrap.cc:492] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:40.430299 23488 ts_tablet_manager.cc:1367] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 14:15:40.430464 23491 raft_consensus.cc:729] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:40.430737 23491 consensus_queue.cc:260] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:40.431495 23488 raft_consensus.cc:348] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:40.431777 23488 raft_consensus.cc:374] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:40.431882 23488 raft_consensus.cc:729] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:40.432168 23488 consensus_queue.cc:260] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:40.432500 23487 tablet_bootstrap.cc:492] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:40.432689 23487 ts_tablet_manager.cc:1367] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240502 14:15:40.433594 23491 ts_tablet_manager.cc:1398] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:40.433660 23487 raft_consensus.cc:348] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:40.434051 23487 raft_consensus.cc:374] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:40.434193 23487 raft_consensus.cc:729] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:40.434499 23487 consensus_queue.cc:260] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:40.435811 23487 ts_tablet_manager.cc:1398] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:40.436206 23487 tablet_bootstrap.cc:492] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:40.438788 23487 tablet_bootstrap.cc:654] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:40.440819 23487 tablet_bootstrap.cc:492] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:40.441011 23487 ts_tablet_manager.cc:1367] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240502 14:15:40.441780 23487 raft_consensus.cc:348] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:40.441900 23488 ts_tablet_manager.cc:1398] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20240502 14:15:40.441967 23487 raft_consensus.cc:374] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:40.442055 23487 raft_consensus.cc:729] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:40.442191 23490 tablet_bootstrap.cc:492] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:40.442315 23490 ts_tablet_manager.cc:1367] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20240502 14:15:40.442286 23487 consensus_queue.cc:260] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:40.442615 23488 tablet_bootstrap.cc:492] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:40.443276 23487 ts_tablet_manager.cc:1398] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:40.443567 23490 raft_consensus.cc:348] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:40.443809 23490 raft_consensus.cc:374] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:40.443928 23490 raft_consensus.cc:729] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:40.444120 23488 tablet_bootstrap.cc:654] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:40.444205 23490 consensus_queue.cc:260] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:40.445380 23490 ts_tablet_manager.cc:1398] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:40.445642 23490 tablet_bootstrap.cc:492] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:40.447230 23490 tablet_bootstrap.cc:654] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:40.447227 23488 tablet_bootstrap.cc:492] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:40.447383 23488 ts_tablet_manager.cc:1367] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20240502 14:15:40.448107 23488 raft_consensus.cc:348] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:40.448302 23488 raft_consensus.cc:374] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:40.448374 23488 raft_consensus.cc:729] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:40.448583 23488 consensus_queue.cc:260] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:40.449033 23488 ts_tablet_manager.cc:1398] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:40.449339 23488 tablet_bootstrap.cc:492] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:40.450096 23490 tablet_bootstrap.cc:492] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:40.450281 23490 ts_tablet_manager.cc:1367] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:15:40.450697 23488 tablet_bootstrap.cc:654] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:40.450945 23490 raft_consensus.cc:348] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:40.451170 23490 raft_consensus.cc:374] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:40.451283 23490 raft_consensus.cc:729] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:40.451536 23490 consensus_queue.cc:260] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:40.452097 23490 ts_tablet_manager.cc:1398] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:40.452375 23488 tablet_bootstrap.cc:492] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:40.452490 23488 ts_tablet_manager.cc:1367] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:40.452999 23488 raft_consensus.cc:348] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:40.453153 23488 raft_consensus.cc:374] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:40.453274 23488 raft_consensus.cc:729] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:40.453490 23488 consensus_queue.cc:260] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:40.453908 23488 ts_tablet_manager.cc:1398] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:15:40.493165 23343 raft_consensus.cc:482] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:40.493355 23343 raft_consensus.cc:504] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:40.494130 23343 leader_election.cc:288] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:40.494412 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:40.494607 22311 raft_consensus.cc:2454] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:40.495116 22640 leader_election.cc:302] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:40.495309 23343 raft_consensus.cc:2781] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:40.495404 23343 raft_consensus.cc:482] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:40.495497 23343 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:40.497149 23343 raft_consensus.cc:504] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:40.497766 23343 leader_election.cc:288] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:40.498003 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:40.498157 22311 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:40.499680 22311 raft_consensus.cc:2454] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:40.499790 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:40.499935 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:40.500059 22445 raft_consensus.cc:2454] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:40.500072 22640 leader_election.cc:302] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:40.500341 23343 raft_consensus.cc:2781] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:40.500490 23343 raft_consensus.cc:686] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:40.500759 23343 consensus_queue.cc:237] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:40.503060 22166 catalog_manager.cc:5484] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:40.534857 23496 raft_consensus.cc:482] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:40.534996 23496 raft_consensus.cc:504] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:40.535562 23343 raft_consensus.cc:482] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:40.535724 23343 raft_consensus.cc:504] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:40.536257 23343 leader_election.cc:288] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:40.536576 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:40.536665 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242" is_pre_election: true
I20240502 14:15:40.536800 22444 raft_consensus.cc:2454] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:40.536892 22576 raft_consensus.cc:2454] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:40.537438 22642 leader_election.cc:302] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:40.537778 23501 raft_consensus.cc:2781] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:40.537869 23501 raft_consensus.cc:482] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:40.537983 23501 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:40.541620 23496 leader_election.cc:288] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:40.541747 23501 raft_consensus.cc:504] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:40.542943 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:40.543205 22576 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:40.547503 22576 raft_consensus.cc:2454] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:40.547952 22642 leader_election.cc:302] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:40.548534 23501 leader_election.cc:288] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:40.548624 23509 raft_consensus.cc:2781] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:40.548916 23509 raft_consensus.cc:686] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:40.549513 23509 consensus_queue.cc:237] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:40.551167 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:40.551343 22444 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:40.554194 22444 raft_consensus.cc:2454] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:40.556733 22166 catalog_manager.cc:5484] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:40.558701 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:40.560075 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242" is_pre_election: true
I20240502 14:15:40.560266 22576 raft_consensus.cc:2379] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 in current term 1: Already voted for candidate c8a8e31da9734e118e5095919ae17f10 in this term.
I20240502 14:15:40.560667 22379 leader_election.cc:302] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5; no voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:40.560950 23496 raft_consensus.cc:2726] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240502 14:15:40.623430 23509 raft_consensus.cc:482] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:40.623592 23509 raft_consensus.cc:504] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:40.624192 23509 leader_election.cc:288] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:40.624614 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:40.624647 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:40.624783 22444 raft_consensus.cc:2454] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:40.624843 22311 raft_consensus.cc:2454] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:40.625237 22643 leader_election.cc:302] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:40.625497 23509 raft_consensus.cc:2781] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:40.625567 23509 raft_consensus.cc:482] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:40.625630 23509 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:40.627135 23509 raft_consensus.cc:504] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:40.627544 23509 leader_election.cc:288] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:40.627892 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:40.627931 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:40.628036 22444 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:40.628100 22311 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:40.629899 22311 raft_consensus.cc:2454] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:40.629899 22444 raft_consensus.cc:2454] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:40.630200 22640 leader_election.cc:302] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:40.630452 23509 raft_consensus.cc:2781] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:40.630611 23509 raft_consensus.cc:686] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:40.630836 23509 consensus_queue.cc:237] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:40.632958 22166 catalog_manager.cc:5484] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:40.884855 23509 consensus_queue.cc:1035] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:40.889312 22444 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:40.893676 23343 consensus_queue.cc:1035] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:15:40.931674 23512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:40.931934 23512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:40.938485 23512 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240502 14:15:41.054728 23343 consensus_queue.cc:1035] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:41.066566 23501 consensus_queue.cc:1035] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:41.086824 22312 raft_consensus.cc:1261] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:41.087284 23501 consensus_queue.cc:1035] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:41.090749 22443 raft_consensus.cc:1261] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:41.098392 23509 consensus_queue.cc:1035] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:15:41.229146 22643 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.194:39183, user_credentials={real_user=slave}} blocked reactor thread for 54875.8us
INSERT report
rows total: 2000
time total: 261.3 ms
time per row: 0.13065 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:15:42.161005 22165 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:42.169890 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 33901f8df4ce435ca5c953d31eddcb28 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d133c835ba274cd79efc8c519436b2aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:42.170349 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33901f8df4ce435ca5c953d31eddcb28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:42.171747 22685 tablet_service.cc:1467] Processing CreateTablet for tablet 34e213ccb9cf4cb7a4a3526e5af74777 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d133c835ba274cd79efc8c519436b2aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:42.171694 22688 tablet_service.cc:1467] Processing CreateTablet for tablet a53d97bb173340c586fe577b79ca5a55 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d133c835ba274cd79efc8c519436b2aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:42.172137 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e213ccb9cf4cb7a4a3526e5af74777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:42.172178 22422 tablet_service.cc:1467] Processing CreateTablet for tablet 34e213ccb9cf4cb7a4a3526e5af74777 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d133c835ba274cd79efc8c519436b2aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:42.172477 22688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53d97bb173340c586fe577b79ca5a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:42.172569 22422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e213ccb9cf4cb7a4a3526e5af74777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:42.176190 22424 tablet_service.cc:1467] Processing CreateTablet for tablet 33901f8df4ce435ca5c953d31eddcb28 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d133c835ba274cd79efc8c519436b2aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:42.176499 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33901f8df4ce435ca5c953d31eddcb28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:42.177865 22292 tablet_service.cc:1467] Processing CreateTablet for tablet 33901f8df4ce435ca5c953d31eddcb28 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d133c835ba274cd79efc8c519436b2aa]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:42.178309 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33901f8df4ce435ca5c953d31eddcb28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:42.179183 23575 tablet_bootstrap.cc:492] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:42.180346 23575 tablet_bootstrap.cc:654] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:42.181458 22290 tablet_service.cc:1467] Processing CreateTablet for tablet a53d97bb173340c586fe577b79ca5a55 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d133c835ba274cd79efc8c519436b2aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:42.181833 22290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53d97bb173340c586fe577b79ca5a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:42.181892 22555 tablet_service.cc:1467] Processing CreateTablet for tablet a53d97bb173340c586fe577b79ca5a55 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d133c835ba274cd79efc8c519436b2aa]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:42.182085 23575 tablet_bootstrap.cc:492] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:42.182229 22555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a53d97bb173340c586fe577b79ca5a55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:42.182220 23575 ts_tablet_manager.cc:1367] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:42.182874 23575 raft_consensus.cc:348] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.183101 23575 raft_consensus.cc:374] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:42.183199 23575 raft_consensus.cc:729] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:42.183406 23575 consensus_queue.cc:260] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.183905 23575 ts_tablet_manager.cc:1398] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:42.185581 22554 tablet_service.cc:1467] Processing CreateTablet for tablet 34e213ccb9cf4cb7a4a3526e5af74777 (DEFAULT_TABLE table=rebalance_test_table_3 [id=d133c835ba274cd79efc8c519436b2aa]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:42.185958 22554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e213ccb9cf4cb7a4a3526e5af74777. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:42.191231 23577 tablet_bootstrap.cc:492] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:42.192804 23577 tablet_bootstrap.cc:654] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:42.199484 23574 tablet_bootstrap.cc:492] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:42.201603 23574 tablet_bootstrap.cc:654] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:42.202807 23579 tablet_bootstrap.cc:492] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:42.203847 23575 tablet_bootstrap.cc:492] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:42.205547 23579 tablet_bootstrap.cc:654] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:42.206010 23575 tablet_bootstrap.cc:654] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:42.207007 23577 tablet_bootstrap.cc:492] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:42.207208 23577 ts_tablet_manager.cc:1367] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20240502 14:15:42.207661 23574 tablet_bootstrap.cc:492] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:42.207898 23574 ts_tablet_manager.cc:1367] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 14:15:42.208005 23577 raft_consensus.cc:348] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.208282 23577 raft_consensus.cc:374] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:42.208405 23577 raft_consensus.cc:729] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:42.208691 23577 consensus_queue.cc:260] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.208884 23574 raft_consensus.cc:348] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.209203 23574 raft_consensus.cc:374] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:42.209385 23574 raft_consensus.cc:729] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:42.209743 23574 consensus_queue.cc:260] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.213441 23574 ts_tablet_manager.cc:1398] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:15:42.213732 23574 tablet_bootstrap.cc:492] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:42.215225 23574 tablet_bootstrap.cc:654] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:42.215530 23579 tablet_bootstrap.cc:492] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:42.215703 23579 ts_tablet_manager.cc:1367] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20240502 14:15:42.216114 23577 ts_tablet_manager.cc:1398] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.009s user 0.003s sys 0.002s
I20240502 14:15:42.216375 23577 tablet_bootstrap.cc:492] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:42.216559 23575 tablet_bootstrap.cc:492] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:42.216495 23579 raft_consensus.cc:348] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.216769 23579 raft_consensus.cc:374] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:42.216784 23575 ts_tablet_manager.cc:1367] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.008s
I20240502 14:15:42.216904 23579 raft_consensus.cc:729] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:42.217198 23579 consensus_queue.cc:260] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.217461 23575 raft_consensus.cc:348] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:42.217710 23575 raft_consensus.cc:374] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:42.217821 23575 raft_consensus.cc:729] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:42.218259 23577 tablet_bootstrap.cc:654] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:42.219273 23579 ts_tablet_manager.cc:1398] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:42.219671 23575 consensus_queue.cc:260] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:42.221987 23575 ts_tablet_manager.cc:1398] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:42.225068 23574 tablet_bootstrap.cc:492] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:42.225211 23574 ts_tablet_manager.cc:1367] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20240502 14:15:42.225448 23579 tablet_bootstrap.cc:492] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:42.225975 23574 raft_consensus.cc:348] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:42.226244 23574 raft_consensus.cc:374] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:42.226359 23574 raft_consensus.cc:729] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:42.226639 23574 consensus_queue.cc:260] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:42.227574 23579 tablet_bootstrap.cc:654] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:42.228071 23574 ts_tablet_manager.cc:1398] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:15:42.228395 23574 tablet_bootstrap.cc:492] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:42.231418 23574 tablet_bootstrap.cc:654] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:42.232949 23577 tablet_bootstrap.cc:492] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:42.233127 23577 ts_tablet_manager.cc:1367] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20240502 14:15:42.233304 23574 tablet_bootstrap.cc:492] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:42.233366 23579 tablet_bootstrap.cc:492] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:42.233445 23574 ts_tablet_manager.cc:1367] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240502 14:15:42.233547 23579 ts_tablet_manager.cc:1367] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20240502 14:15:42.233809 23577 raft_consensus.cc:348] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:42.233990 23577 raft_consensus.cc:374] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:42.234069 23577 raft_consensus.cc:729] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:42.234134 23574 raft_consensus.cc:348] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.234194 23579 raft_consensus.cc:348] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.234367 23574 raft_consensus.cc:374] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:42.234318 23577 consensus_queue.cc:260] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:42.234457 23574 raft_consensus.cc:729] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:42.234472 23579 raft_consensus.cc:374] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:42.234536 23579 raft_consensus.cc:729] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:42.234650 23574 consensus_queue.cc:260] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.234916 23577 ts_tablet_manager.cc:1398] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:42.235175 23574 ts_tablet_manager.cc:1398] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:42.235467 23579 consensus_queue.cc:260] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.235913 23579 ts_tablet_manager.cc:1398] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:42.246331 23564 raft_consensus.cc:482] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:42.246503 23564 raft_consensus.cc:504] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:42.247242 23564 leader_election.cc:288] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:42.247581 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242" is_pre_election: true
I20240502 14:15:42.247767 22575 raft_consensus.cc:2454] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:42.247713 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:42.247967 22445 raft_consensus.cc:2454] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:42.248188 22642 leader_election.cc:302] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:42.248494 23564 raft_consensus.cc:2781] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:42.248615 23564 raft_consensus.cc:482] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:42.248692 23564 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:42.250499 23564 raft_consensus.cc:504] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:42.251050 23564 leader_election.cc:288] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:42.251394 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:42.251545 22575 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:42.251492 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:42.251705 22445 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:42.253512 22575 raft_consensus.cc:2454] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:42.253747 22445 raft_consensus.cc:2454] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:42.253906 22642 leader_election.cc:302] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:42.254232 23564 raft_consensus.cc:2781] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:42.254438 23564 raft_consensus.cc:686] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:42.254678 23564 consensus_queue.cc:237] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:42.257421 22165 catalog_manager.cc:5484] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:42.272166 23342 raft_consensus.cc:482] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:42.272281 23342 raft_consensus.cc:504] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.272795 23342 leader_election.cc:288] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:42.273155 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53d97bb173340c586fe577b79ca5a55" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:42.273365 22708 raft_consensus.cc:2454] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:42.273344 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53d97bb173340c586fe577b79ca5a55" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:42.273543 22311 raft_consensus.cc:2454] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:42.273777 22509 leader_election.cc:302] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:42.274008 23342 raft_consensus.cc:2781] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:42.274125 23342 raft_consensus.cc:482] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:42.274231 23342 raft_consensus.cc:3037] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:42.276028 23342 raft_consensus.cc:504] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.276614 23342 leader_election.cc:288] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:42.276903 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53d97bb173340c586fe577b79ca5a55" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:42.277051 22708 raft_consensus.cc:3037] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:42.277036 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53d97bb173340c586fe577b79ca5a55" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:42.277263 22311 raft_consensus.cc:3037] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:42.278899 22708 raft_consensus.cc:2454] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:42.279142 22311 raft_consensus.cc:2454] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:42.279215 22509 leader_election.cc:302] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:42.279476 23342 raft_consensus.cc:2781] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:42.279814 23342 raft_consensus.cc:686] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:42.280078 23342 consensus_queue.cc:237] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.282261 22165 catalog_manager.cc:5484] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:42.358989 23342 raft_consensus.cc:482] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:42.359221 23342 raft_consensus.cc:504] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.359804 23342 leader_election.cc:288] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:42.360334 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:42.360379 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:42.360592 22445 raft_consensus.cc:2454] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:42.360582 22311 raft_consensus.cc:2454] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:42.361001 22509 leader_election.cc:302] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:42.361277 23342 raft_consensus.cc:2781] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:42.361379 23342 raft_consensus.cc:482] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:42.361490 23342 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:42.363341 23342 raft_consensus.cc:504] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.363894 23342 leader_election.cc:288] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:42.364195 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:42.364367 22445 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:42.364315 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:42.364504 22311 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:42.366379 22311 raft_consensus.cc:2454] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:42.366410 22445 raft_consensus.cc:2454] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:42.366725 22509 leader_election.cc:302] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:42.367007 23342 raft_consensus.cc:2781] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:42.367149 23342 raft_consensus.cc:686] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:42.367422 23342 consensus_queue.cc:237] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:42.370043 22165 catalog_manager.cc:5484] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:42.659668 23342 consensus_queue.cc:1035] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:42.666244 23342 consensus_queue.cc:1035] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:15:42.684021 23593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:42.684288 23593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:42.690562 23593 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240502 14:15:42.796182 23564 consensus_queue.cc:1035] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:42.818630 22444 raft_consensus.cc:1261] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:42.819263 23342 consensus_queue.cc:1035] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:42.834303 22575 raft_consensus.cc:1261] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:42.834825 23564 consensus_queue.cc:1035] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:42.836624 23590 consensus_queue.cc:1035] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 219.893 ms
time per row: 0.109947 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:15:43.902222 22165 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:43.910358 22292 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef6bc006a79451babe394ba112dd717 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c9158a710cb3447bb08d3371c5306842]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:43.910640 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef6bc006a79451babe394ba112dd717 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c9158a710cb3447bb08d3371c5306842]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:43.910846 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef6bc006a79451babe394ba112dd717. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:43.910760 22422 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef6bc006a79451babe394ba112dd717 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c9158a710cb3447bb08d3371c5306842]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:43.911092 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef6bc006a79451babe394ba112dd717. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:43.911145 22422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef6bc006a79451babe394ba112dd717. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:43.912631 22555 tablet_service.cc:1467] Processing CreateTablet for tablet f9da0fcab2294a3fb62c132a6b60214c (DEFAULT_TABLE table=rebalance_test_table_4 [id=c9158a710cb3447bb08d3371c5306842]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:43.912942 22555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9da0fcab2294a3fb62c132a6b60214c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:43.913570 22291 tablet_service.cc:1467] Processing CreateTablet for tablet f9da0fcab2294a3fb62c132a6b60214c (DEFAULT_TABLE table=rebalance_test_table_4 [id=c9158a710cb3447bb08d3371c5306842]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:43.913914 22291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9da0fcab2294a3fb62c132a6b60214c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:43.917301 22685 tablet_service.cc:1467] Processing CreateTablet for tablet f9da0fcab2294a3fb62c132a6b60214c (DEFAULT_TABLE table=rebalance_test_table_4 [id=c9158a710cb3447bb08d3371c5306842]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:43.917340 22290 tablet_service.cc:1467] Processing CreateTablet for tablet 64ec9e34ad4d49faad727155411f9762 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c9158a710cb3447bb08d3371c5306842]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:43.917393 22688 tablet_service.cc:1467] Processing CreateTablet for tablet 64ec9e34ad4d49faad727155411f9762 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c9158a710cb3447bb08d3371c5306842]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:43.917627 22290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ec9e34ad4d49faad727155411f9762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:43.917691 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9da0fcab2294a3fb62c132a6b60214c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:43.918218 22688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ec9e34ad4d49faad727155411f9762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:43.922709 22424 tablet_service.cc:1467] Processing CreateTablet for tablet 64ec9e34ad4d49faad727155411f9762 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c9158a710cb3447bb08d3371c5306842]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:43.923035 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64ec9e34ad4d49faad727155411f9762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:43.930079 23667 tablet_bootstrap.cc:492] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:43.931970 23667 tablet_bootstrap.cc:654] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:43.935076 23665 tablet_bootstrap.cc:492] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:43.936435 23665 tablet_bootstrap.cc:654] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:43.937938 23668 tablet_bootstrap.cc:492] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:43.938237 23665 tablet_bootstrap.cc:492] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:43.938686 23665 ts_tablet_manager.cc:1367] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:15:43.939307 23668 tablet_bootstrap.cc:654] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:43.940423 23666 tablet_bootstrap.cc:492] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:43.940840 23665 raft_consensus.cc:348] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.941141 23665 raft_consensus.cc:374] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:43.941296 23665 raft_consensus.cc:729] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:43.941581 23668 tablet_bootstrap.cc:492] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:43.941609 23665 consensus_queue.cc:260] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.941810 23668 ts_tablet_manager.cc:1367] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240502 14:15:43.942670 23666 tablet_bootstrap.cc:654] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:43.942699 23668 raft_consensus.cc:348] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:43.943020 23668 raft_consensus.cc:374] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:43.943178 23668 raft_consensus.cc:729] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:43.943573 23668 consensus_queue.cc:260] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:43.945204 23668 ts_tablet_manager.cc:1398] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:43.945752 23668 tablet_bootstrap.cc:492] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:43.947003 23665 ts_tablet_manager.cc:1398] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.008s user 0.001s sys 0.003s
I20240502 14:15:43.947428 23665 tablet_bootstrap.cc:492] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:43.948014 23668 tablet_bootstrap.cc:654] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:43.949344 23665 tablet_bootstrap.cc:654] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:43.953511 23666 tablet_bootstrap.cc:492] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:43.953611 23667 tablet_bootstrap.cc:492] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:43.953707 23666 ts_tablet_manager.cc:1367] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20240502 14:15:43.953814 23667 ts_tablet_manager.cc:1367] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.024s user 0.015s sys 0.000s
I20240502 14:15:43.954316 23668 tablet_bootstrap.cc:492] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:43.954329 23666 raft_consensus.cc:348] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.954483 23668 ts_tablet_manager.cc:1367] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240502 14:15:43.954525 23666 raft_consensus.cc:374] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:43.954610 23666 raft_consensus.cc:729] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:43.954609 23667 raft_consensus.cc:348] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:43.954793 23667 raft_consensus.cc:374] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:43.954799 23666 consensus_queue.cc:260] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.954890 23667 raft_consensus.cc:729] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:43.955075 23667 consensus_queue.cc:260] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:43.955106 23668 raft_consensus.cc:348] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.955281 23668 raft_consensus.cc:374] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:43.955349 23668 raft_consensus.cc:729] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:43.955543 23668 consensus_queue.cc:260] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.955749 23666 ts_tablet_manager.cc:1398] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:43.955998 23666 tablet_bootstrap.cc:492] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:43.956226 23665 tablet_bootstrap.cc:492] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:43.956370 23665 ts_tablet_manager.cc:1367] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240502 14:15:43.957098 23665 raft_consensus.cc:348] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.957319 23665 raft_consensus.cc:374] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:43.957389 23665 raft_consensus.cc:729] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:43.957398 23667 ts_tablet_manager.cc:1398] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:43.957598 23665 consensus_queue.cc:260] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.958065 23665 ts_tablet_manager.cc:1398] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:43.958673 23667 tablet_bootstrap.cc:492] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:43.958909 23666 tablet_bootstrap.cc:654] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:43.959034 23668 ts_tablet_manager.cc:1398] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240502 14:15:43.960048 23667 tablet_bootstrap.cc:654] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:43.962507 23666 tablet_bootstrap.cc:492] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:43.962715 23666 ts_tablet_manager.cc:1367] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20240502 14:15:43.963092 23667 tablet_bootstrap.cc:492] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:43.963264 23667 ts_tablet_manager.cc:1367] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240502 14:15:43.963464 23666 raft_consensus.cc:348] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:43.963708 23666 raft_consensus.cc:374] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:43.963953 23666 raft_consensus.cc:729] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:43.964150 23667 raft_consensus.cc:348] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.964401 23667 raft_consensus.cc:374] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:43.964237 23666 consensus_queue.cc:260] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:43.964493 23667 raft_consensus.cc:729] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:43.964937 23666 ts_tablet_manager.cc:1398] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:43.964807 23667 consensus_queue.cc:260] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.965516 23667 ts_tablet_manager.cc:1398] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:43.965782 23667 tablet_bootstrap.cc:492] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:43.966817 23667 tablet_bootstrap.cc:654] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:43.968742 23667 tablet_bootstrap.cc:492] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:43.968899 23667 ts_tablet_manager.cc:1367] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:43.969592 23667 raft_consensus.cc:348] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.969770 23667 raft_consensus.cc:374] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:43.969907 23667 raft_consensus.cc:729] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:43.970104 23667 consensus_queue.cc:260] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.970585 23667 ts_tablet_manager.cc:1398] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:43.975927 23672 raft_consensus.cc:482] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:43.976055 23672 raft_consensus.cc:504] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.976711 23672 leader_election.cc:288] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:43.977751 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:43.977979 22708 raft_consensus.cc:2454] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 0.
I20240502 14:15:43.978432 22377 leader_election.cc:302] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:43.978761 23672 raft_consensus.cc:2781] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:43.978888 23672 raft_consensus.cc:482] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:43.978984 23672 raft_consensus.cc:3037] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:43.980989 23672 raft_consensus.cc:504] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.981515 23672 leader_election.cc:288] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:43.981978 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:43.982157 22708 raft_consensus.cc:3037] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:43.983371 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:43.983536 22311 raft_consensus.cc:2454] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 0.
I20240502 14:15:43.983505 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:43.983682 22312 raft_consensus.cc:3037] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:43.984678 22708 raft_consensus.cc:2454] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 1.
I20240502 14:15:43.985069 22377 leader_election.cc:302] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:43.985325 23672 raft_consensus.cc:2781] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:43.985698 23672 raft_consensus.cc:686] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:43.986269 22312 raft_consensus.cc:2454] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 1.
I20240502 14:15:43.986167 23672 consensus_queue.cc:237] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:43.989180 22169 catalog_manager.cc:5484] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 1 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:44.017558 23640 raft_consensus.cc:482] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:44.017709 23640 raft_consensus.cc:504] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:44.018306 23640 leader_election.cc:288] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:44.018808 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9da0fcab2294a3fb62c132a6b60214c" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:44.018908 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9da0fcab2294a3fb62c132a6b60214c" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:44.019023 22708 raft_consensus.cc:2454] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:44.019165 22312 raft_consensus.cc:2454] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:44.019443 22509 leader_election.cc:302] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:44.019732 23640 raft_consensus.cc:2781] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:44.019878 23640 raft_consensus.cc:482] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:44.019971 23640 raft_consensus.cc:3037] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:44.022168 23640 raft_consensus.cc:504] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:44.022608 23640 leader_election.cc:288] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:44.023003 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9da0fcab2294a3fb62c132a6b60214c" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:44.023072 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9da0fcab2294a3fb62c132a6b60214c" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:44.023175 22708 raft_consensus.cc:3037] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:44.023275 22312 raft_consensus.cc:3037] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:44.024974 22708 raft_consensus.cc:2454] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:44.025061 22312 raft_consensus.cc:2454] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:44.025301 22509 leader_election.cc:302] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:44.025702 23640 raft_consensus.cc:2781] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:44.026160 23640 raft_consensus.cc:686] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:44.026443 23640 consensus_queue.cc:237] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:44.028918 22169 catalog_manager.cc:5484] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:44.078827 23640 raft_consensus.cc:482] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:44.078994 23640 raft_consensus.cc:504] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:44.079479 23640 leader_election.cc:288] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:44.079928 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:44.079932 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:44.080158 22444 raft_consensus.cc:2454] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:44.080171 22312 raft_consensus.cc:2454] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:44.080551 22509 leader_election.cc:302] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:44.080821 23640 raft_consensus.cc:2781] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:44.080919 23640 raft_consensus.cc:482] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:44.081004 23640 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:44.082690 23640 raft_consensus.cc:504] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:44.083151 23640 leader_election.cc:288] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:44.083539 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:44.083586 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:44.083706 22312 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:44.083746 22444 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:44.085445 22312 raft_consensus.cc:2454] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:44.085510 22444 raft_consensus.cc:2454] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:44.085794 22509 leader_election.cc:302] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:44.086020 23640 raft_consensus.cc:2781] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:44.086160 23640 raft_consensus.cc:686] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:44.086380 23640 consensus_queue.cc:237] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:44.088639 22169 catalog_manager.cc:5484] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:44.425669 23686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:44.425945 23686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:44.432925 23686 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240502 14:15:44.550902 22708 raft_consensus.cc:1261] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:44.553050 23723 consensus_queue.cc:1035] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:44.556695 22312 raft_consensus.cc:1261] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:44.557341 23640 consensus_queue.cc:1035] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:44.556195 22444 raft_consensus.cc:1261] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:44.558949 23683 consensus_queue.cc:1035] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:44.561136 22311 raft_consensus.cc:1261] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:44.561839 22708 raft_consensus.cc:1261] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:44.562352 23683 consensus_queue.cc:1035] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:44.562738 23683 consensus_queue.cc:1035] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:44.593176 22310 raft_consensus.cc:1261] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:44.597728 23725 consensus_queue.cc:1035] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 257.726 ms
time per row: 0.128863 ms
I20240502 14:15:44.823719 22162 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:44.831876 22422 tablet_service.cc:1467] Processing CreateTablet for tablet 2a5c477465164ee2aa6d679aff875079 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8691edc1ee34f609390e61d2829a813]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:44.832118 22685 tablet_service.cc:1467] Processing CreateTablet for tablet 2a5c477465164ee2aa6d679aff875079 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8691edc1ee34f609390e61d2829a813]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:44.832342 22422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a5c477465164ee2aa6d679aff875079. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:44.832512 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a5c477465164ee2aa6d679aff875079. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:44.833503 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 2a5c477465164ee2aa6d679aff875079 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8691edc1ee34f609390e61d2829a813]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:44.833676 22554 tablet_service.cc:1467] Processing CreateTablet for tablet cd6972bee4d94830ac2d6a5660adc944 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8691edc1ee34f609390e61d2829a813]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:44.833949 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a5c477465164ee2aa6d679aff875079. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:44.834050 22424 tablet_service.cc:1467] Processing CreateTablet for tablet 1e4e12e267d2482d83dfe8c14f942d25 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8691edc1ee34f609390e61d2829a813]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:44.834214 22554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6972bee4d94830ac2d6a5660adc944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:44.834393 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e4e12e267d2482d83dfe8c14f942d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:44.837975 22555 tablet_service.cc:1467] Processing CreateTablet for tablet 1e4e12e267d2482d83dfe8c14f942d25 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8691edc1ee34f609390e61d2829a813]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:44.838387 22555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e4e12e267d2482d83dfe8c14f942d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:44.840094 22423 tablet_service.cc:1467] Processing CreateTablet for tablet cd6972bee4d94830ac2d6a5660adc944 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8691edc1ee34f609390e61d2829a813]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:44.840520 22423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6972bee4d94830ac2d6a5660adc944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:44.841643 22290 tablet_service.cc:1467] Processing CreateTablet for tablet cd6972bee4d94830ac2d6a5660adc944 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8691edc1ee34f609390e61d2829a813]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:44.842018 22290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd6972bee4d94830ac2d6a5660adc944. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:44.846567 22688 tablet_service.cc:1467] Processing CreateTablet for tablet 1e4e12e267d2482d83dfe8c14f942d25 (DEFAULT_TABLE table=rebalance_test_table_5 [id=c8691edc1ee34f609390e61d2829a813]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:44.847010 22688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e4e12e267d2482d83dfe8c14f942d25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:44.850518 23760 tablet_bootstrap.cc:492] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:44.852372 23760 tablet_bootstrap.cc:654] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:44.854343 23761 tablet_bootstrap.cc:492] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:44.855887 23761 tablet_bootstrap.cc:654] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:44.857633 23760 tablet_bootstrap.cc:492] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:44.857800 23760 ts_tablet_manager.cc:1367] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240502 14:15:44.857831 23764 tablet_bootstrap.cc:492] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:44.858860 23761 tablet_bootstrap.cc:492] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:44.858836 23760 raft_consensus.cc:348] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:44.859066 23761 ts_tablet_manager.cc:1367] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240502 14:15:44.859113 23760 raft_consensus.cc:374] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:44.859254 23760 raft_consensus.cc:729] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:44.859673 23764 tablet_bootstrap.cc:654] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:44.859560 23760 consensus_queue.cc:260] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:44.859901 23761 raft_consensus.cc:348] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:44.860206 23761 raft_consensus.cc:374] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:44.860350 23761 raft_consensus.cc:729] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:44.860819 23760 ts_tablet_manager.cc:1398] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240502 14:15:44.861212 23760 tablet_bootstrap.cc:492] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:44.861034 23761 consensus_queue.cc:260] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:44.863605 23761 ts_tablet_manager.cc:1398] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240502 14:15:44.864002 23761 tablet_bootstrap.cc:492] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:44.864188 23760 tablet_bootstrap.cc:654] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:44.865319 23765 tablet_bootstrap.cc:492] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:44.866478 23761 tablet_bootstrap.cc:654] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:44.867275 23765 tablet_bootstrap.cc:654] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:44.872973 23760 tablet_bootstrap.cc:492] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:44.873209 23760 ts_tablet_manager.cc:1367] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240502 14:15:44.874279 23760 raft_consensus.cc:348] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:44.874563 23764 tablet_bootstrap.cc:492] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:44.874581 23760 raft_consensus.cc:374] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:44.874653 23760 raft_consensus.cc:729] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:44.874917 23764 ts_tablet_manager.cc:1367] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.007s
I20240502 14:15:44.874889 23760 consensus_queue.cc:260] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:44.875305 23765 tablet_bootstrap.cc:492] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:44.875476 23765 ts_tablet_manager.cc:1367] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240502 14:15:44.875919 23760 ts_tablet_manager.cc:1398] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:44.876180 23760 tablet_bootstrap.cc:492] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:44.876315 23765 raft_consensus.cc:348] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:44.876544 23765 raft_consensus.cc:374] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:44.876659 23765 raft_consensus.cc:729] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:44.876986 23765 consensus_queue.cc:260] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:44.877768 23760 tablet_bootstrap.cc:654] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:44.877782 23765 ts_tablet_manager.cc:1398] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:44.880375 23764 raft_consensus.cc:348] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:44.881062 23764 raft_consensus.cc:374] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:44.881968 23764 raft_consensus.cc:729] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:44.882361 23764 consensus_queue.cc:260] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:44.883041 23764 ts_tablet_manager.cc:1398] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240502 14:15:44.883975 23764 tablet_bootstrap.cc:492] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:44.885440 23760 tablet_bootstrap.cc:492] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:44.885557 23760 ts_tablet_manager.cc:1367] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.001s
I20240502 14:15:44.885645 23764 tablet_bootstrap.cc:654] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:44.886137 23760 raft_consensus.cc:348] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:44.886269 23760 raft_consensus.cc:374] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:44.886349 23760 raft_consensus.cc:729] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:44.886533 23760 consensus_queue.cc:260] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:44.887125 23760 ts_tablet_manager.cc:1398] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240502 14:15:44.887370 23761 tablet_bootstrap.cc:492] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:44.887567 23761 ts_tablet_manager.cc:1367] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.024s user 0.012s sys 0.001s
I20240502 14:15:44.888291 23761 raft_consensus.cc:348] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:44.888515 23761 raft_consensus.cc:374] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:44.888600 23761 raft_consensus.cc:729] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:44.888847 23761 consensus_queue.cc:260] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:44.889355 23761 ts_tablet_manager.cc:1398] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:44.889670 23761 tablet_bootstrap.cc:492] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:44.891250 23761 tablet_bootstrap.cc:654] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:44.892921 23761 tablet_bootstrap.cc:492] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:44.893038 23761 ts_tablet_manager.cc:1367] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:44.893056 23764 tablet_bootstrap.cc:492] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:44.893213 23764 ts_tablet_manager.cc:1367] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240502 14:15:44.893692 23761 raft_consensus.cc:348] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:44.893841 23761 raft_consensus.cc:374] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:44.893930 23761 raft_consensus.cc:729] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:44.893925 23764 raft_consensus.cc:348] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:44.894183 23764 raft_consensus.cc:374] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:44.894129 23761 consensus_queue.cc:260] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:44.894268 23764 raft_consensus.cc:729] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:44.894553 23764 consensus_queue.cc:260] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:44.894670 23761 ts_tablet_manager.cc:1398] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:44.895044 23764 ts_tablet_manager.cc:1398] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:44.914719 23723 raft_consensus.cc:482] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:44.914862 23723 raft_consensus.cc:504] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:44.915380 23723 leader_election.cc:288] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:44.915783 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:44.915783 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242" is_pre_election: true
I20240502 14:15:44.915959 22575 raft_consensus.cc:2454] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 0.
I20240502 14:15:44.915978 22708 raft_consensus.cc:2454] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 0.
I20240502 14:15:44.916327 22379 leader_election.cc:302] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:44.916536 23723 raft_consensus.cc:2781] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:44.916630 23723 raft_consensus.cc:482] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:44.916697 23723 raft_consensus.cc:3037] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:44.918534 23723 raft_consensus.cc:504] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:44.919001 23723 leader_election.cc:288] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:44.919313 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:44.919489 22575 raft_consensus.cc:3037] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:44.919453 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:44.919621 22708 raft_consensus.cc:3037] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:44.922029 22575 raft_consensus.cc:2454] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 1.
I20240502 14:15:44.922096 22708 raft_consensus.cc:2454] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 1.
I20240502 14:15:44.922446 22379 leader_election.cc:302] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:44.922708 23723 raft_consensus.cc:2781] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:44.922817 23723 raft_consensus.cc:686] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:44.923048 23723 consensus_queue.cc:237] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:44.925339 22170 catalog_manager.cc:5484] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 1 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:44.937268 23746 raft_consensus.cc:482] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:44.937366 23746 raft_consensus.cc:504] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:44.937855 23746 leader_election.cc:288] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:44.938118 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:44.938290 22444 raft_consensus.cc:2454] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:44.938253 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:44.938400 22708 raft_consensus.cc:2454] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:44.938589 22512 leader_election.cc:302] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:44.938791 23746 raft_consensus.cc:2781] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:44.938891 23746 raft_consensus.cc:482] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:44.938959 23746 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:44.940568 23746 raft_consensus.cc:504] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:44.941051 23746 leader_election.cc:288] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:44.941339 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:44.941511 22444 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:44.941588 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:44.941756 22708 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:44.943321 22708 raft_consensus.cc:2454] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:44.943684 22509 leader_election.cc:302] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:44.943958 23746 raft_consensus.cc:2781] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:44.944079 23746 raft_consensus.cc:686] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:44.944315 23746 consensus_queue.cc:237] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:44.946666 22170 catalog_manager.cc:5484] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:44.954910 22444 raft_consensus.cc:2454] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:45.154534 23752 raft_consensus.cc:482] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:45.154717 23752 raft_consensus.cc:504] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:45.155382 23752 leader_election.cc:288] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:45.155733 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6972bee4d94830ac2d6a5660adc944" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:45.155887 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6972bee4d94830ac2d6a5660adc944" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:45.155965 22310 raft_consensus.cc:2454] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:45.156095 22444 raft_consensus.cc:2454] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:45.156338 22509 leader_election.cc:302] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:45.156594 23752 raft_consensus.cc:2781] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:45.156713 23752 raft_consensus.cc:482] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:45.156800 23752 raft_consensus.cc:3037] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:45.158646 23752 raft_consensus.cc:504] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:45.159168 23752 leader_election.cc:288] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:45.159442 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6972bee4d94830ac2d6a5660adc944" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:45.159641 22310 raft_consensus.cc:3037] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:45.159636 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6972bee4d94830ac2d6a5660adc944" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:45.159819 22444 raft_consensus.cc:3037] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:45.162544 22310 raft_consensus.cc:2454] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:45.162600 22444 raft_consensus.cc:2454] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:45.162941 22509 leader_election.cc:302] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:45.163228 23752 raft_consensus.cc:2781] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:45.163369 23752 raft_consensus.cc:686] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:45.163619 23752 consensus_queue.cc:237] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:45.166054 22170 catalog_manager.cc:5484] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:45.372965 23752 consensus_queue.cc:1035] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:45.382431 23746 consensus_queue.cc:1035] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:45.395123 23723 consensus_queue.cc:1035] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:45.421720 23725 consensus_queue.cc:1035] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:15:45.534040 23776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:45.534305 23776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:45.540845 23776 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240502 14:15:45.580462 23779 consensus_queue.cc:1035] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:45.585565 23780 consensus_queue.cc:1035] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 225.558 ms
time per row: 0.112779 ms
I20240502 14:15:45.902952 22170 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:45.911433 22423 tablet_service.cc:1467] Processing CreateTablet for tablet 4f84542c71674d618f57fdbf4e5165d5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3bf3e695a8de459f9e9aac4598482590]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:45.911576 22688 tablet_service.cc:1467] Processing CreateTablet for tablet 4f84542c71674d618f57fdbf4e5165d5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3bf3e695a8de459f9e9aac4598482590]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:45.911825 22423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f84542c71674d618f57fdbf4e5165d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:45.911983 22688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f84542c71674d618f57fdbf4e5165d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:45.912904 22424 tablet_service.cc:1467] Processing CreateTablet for tablet 812638aac21c48aab6d9dd72e08d98d4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3bf3e695a8de459f9e9aac4598482590]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:45.912950 22555 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8f2b775fa54c44a5d855d586576696 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3bf3e695a8de459f9e9aac4598482590]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:45.913267 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812638aac21c48aab6d9dd72e08d98d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:45.913331 22555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8f2b775fa54c44a5d855d586576696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:45.915683 22685 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8f2b775fa54c44a5d855d586576696 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3bf3e695a8de459f9e9aac4598482590]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:45.916005 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8f2b775fa54c44a5d855d586576696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:45.920409 22290 tablet_service.cc:1467] Processing CreateTablet for tablet 4f84542c71674d618f57fdbf4e5165d5 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3bf3e695a8de459f9e9aac4598482590]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:45.920917 22290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f84542c71674d618f57fdbf4e5165d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:45.921335 22686 tablet_service.cc:1467] Processing CreateTablet for tablet 812638aac21c48aab6d9dd72e08d98d4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3bf3e695a8de459f9e9aac4598482590]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:45.921618 22686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812638aac21c48aab6d9dd72e08d98d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:45.925676 22291 tablet_service.cc:1467] Processing CreateTablet for tablet 4c8f2b775fa54c44a5d855d586576696 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3bf3e695a8de459f9e9aac4598482590]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:45.926136 22291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c8f2b775fa54c44a5d855d586576696. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:45.926483 23842 tablet_bootstrap.cc:492] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:45.926945 22292 tablet_service.cc:1467] Processing CreateTablet for tablet 812638aac21c48aab6d9dd72e08d98d4 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3bf3e695a8de459f9e9aac4598482590]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:45.927165 23843 tablet_bootstrap.cc:492] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:45.927371 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812638aac21c48aab6d9dd72e08d98d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:45.929821 23843 tablet_bootstrap.cc:654] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:45.930826 23842 tablet_bootstrap.cc:654] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:45.935068 23841 tablet_bootstrap.cc:492] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:45.937038 23846 tablet_bootstrap.cc:492] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:45.937153 23841 tablet_bootstrap.cc:654] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:45.938179 23846 tablet_bootstrap.cc:654] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:45.938985 23843 tablet_bootstrap.cc:492] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:45.939164 23843 ts_tablet_manager.cc:1367] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 14:15:45.939528 23841 tablet_bootstrap.cc:492] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:45.939705 23842 tablet_bootstrap.cc:492] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:45.939749 23841 ts_tablet_manager.cc:1367] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240502 14:15:45.939860 23842 ts_tablet_manager.cc:1367] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.005s
I20240502 14:15:45.940074 23843 raft_consensus.cc:348] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:45.940459 23843 raft_consensus.cc:374] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:45.940605 23843 raft_consensus.cc:729] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:45.940629 23842 raft_consensus.cc:348] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:45.940848 23842 raft_consensus.cc:374] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:45.940697 23841 raft_consensus.cc:348] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:45.940984 23842 raft_consensus.cc:729] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:45.941097 23841 raft_consensus.cc:374] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:45.941030 23843 consensus_queue.cc:260] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:45.941290 23841 raft_consensus.cc:729] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:45.941316 23842 consensus_queue.cc:260] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:45.941586 23841 consensus_queue.cc:260] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:45.942010 23843 ts_tablet_manager.cc:1398] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:45.942283 23843 tablet_bootstrap.cc:492] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:45.942342 23841 ts_tablet_manager.cc:1398] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:45.942628 23841 tablet_bootstrap.cc:492] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:45.943832 23842 ts_tablet_manager.cc:1398] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240502 14:15:45.944242 23843 tablet_bootstrap.cc:654] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:45.944402 23841 tablet_bootstrap.cc:654] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:45.947788 23843 tablet_bootstrap.cc:492] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:45.947973 23843 ts_tablet_manager.cc:1367] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240502 14:15:45.948875 23843 raft_consensus.cc:348] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:45.949165 23843 raft_consensus.cc:374] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:45.949317 23843 raft_consensus.cc:729] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:45.949678 23843 consensus_queue.cc:260] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:45.950392 23843 ts_tablet_manager.cc:1398] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:45.950837 23843 tablet_bootstrap.cc:492] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:45.952736 23843 tablet_bootstrap.cc:654] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:45.954277 23846 tablet_bootstrap.cc:492] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:45.954468 23846 ts_tablet_manager.cc:1367] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.002s
I20240502 14:15:45.955420 23846 raft_consensus.cc:348] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:45.955698 23846 raft_consensus.cc:374] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:45.955791 23846 raft_consensus.cc:729] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:45.956035 23846 consensus_queue.cc:260] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:45.956554 23846 ts_tablet_manager.cc:1398] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:45.957660 23841 tablet_bootstrap.cc:492] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:45.957829 23841 ts_tablet_manager.cc:1367] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.001s
I20240502 14:15:45.958774 23841 raft_consensus.cc:348] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:45.959005 23841 raft_consensus.cc:374] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:45.959103 23841 raft_consensus.cc:729] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:45.959362 23841 consensus_queue.cc:260] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:45.959893 23841 ts_tablet_manager.cc:1398] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:45.960676 23846 tablet_bootstrap.cc:492] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:45.962790 23846 tablet_bootstrap.cc:654] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:45.964730 23843 tablet_bootstrap.cc:492] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:45.964864 23843 ts_tablet_manager.cc:1367] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240502 14:15:45.965638 23843 raft_consensus.cc:348] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:45.965821 23843 raft_consensus.cc:374] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:45.965905 23843 raft_consensus.cc:729] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:45.966127 23843 consensus_queue.cc:260] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:45.966423 23846 tablet_bootstrap.cc:492] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:45.966563 23846 ts_tablet_manager.cc:1367] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240502 14:15:45.966631 23843 ts_tablet_manager.cc:1398] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:45.967259 23846 raft_consensus.cc:348] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:45.967464 23846 raft_consensus.cc:374] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:45.967546 23846 raft_consensus.cc:729] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:45.967751 23846 consensus_queue.cc:260] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:45.968152 23846 ts_tablet_manager.cc:1398] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:45.968410 23846 tablet_bootstrap.cc:492] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:45.969517 23846 tablet_bootstrap.cc:654] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:45.971349 23846 tablet_bootstrap.cc:492] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:45.971494 23846 ts_tablet_manager.cc:1367] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:45.972090 23846 raft_consensus.cc:348] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:45.972223 23846 raft_consensus.cc:374] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:45.972282 23846 raft_consensus.cc:729] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:45.972414 23846 consensus_queue.cc:260] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:45.972730 23846 ts_tablet_manager.cc:1398] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:46.054718 23709 raft_consensus.cc:482] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:46.054857 23709 raft_consensus.cc:504] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:46.055392 23709 leader_election.cc:288] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:46.055688 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:46.055789 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:46.055898 22310 raft_consensus.cc:2454] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:46.056039 22445 raft_consensus.cc:2454] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:46.056195 23320 raft_consensus.cc:482] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:46.056327 23320 raft_consensus.cc:504] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:46.056473 22643 leader_election.cc:302] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:46.056825 23709 raft_consensus.cc:2781] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:46.056929 23320 leader_election.cc:288] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:46.056944 23709 raft_consensus.cc:482] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:46.057021 23709 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:46.057255 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:46.057463 22445 raft_consensus.cc:2454] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 0.
I20240502 14:15:46.057499 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:46.057726 22708 raft_consensus.cc:2454] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 0.
I20240502 14:15:46.057797 22248 leader_election.cc:302] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:46.058089 23320 raft_consensus.cc:2781] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:46.058195 23320 raft_consensus.cc:482] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:46.058291 23320 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:46.059383 23709 raft_consensus.cc:504] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:46.060245 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:46.060312 23320 raft_consensus.cc:504] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:46.060400 22445 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:46.060809 23320 leader_election.cc:288] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:46.061028 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:46.061213 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:46.061295 22310 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:46.061400 22708 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:46.061460 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:46.061640 22444 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:46.063751 22310 raft_consensus.cc:2454] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:46.063788 22708 raft_consensus.cc:2454] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 1.
I20240502 14:15:46.064069 22445 raft_consensus.cc:2454] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:46.064231 22444 raft_consensus.cc:2454] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 1.
I20240502 14:15:46.064206 22245 leader_election.cc:302] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:46.064236 22640 leader_election.cc:302] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:46.064512 23320 raft_consensus.cc:2781] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:46.064541 23787 raft_consensus.cc:2781] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:46.064947 23320 raft_consensus.cc:686] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:46.065372 23709 leader_election.cc:288] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:46.065285 23320 consensus_queue.cc:237] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:46.065847 23787 raft_consensus.cc:686] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:46.066147 23787 consensus_queue.cc:237] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:46.068264 22162 catalog_manager.cc:5484] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:46.068840 22170 catalog_manager.cc:5484] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:46.187156 23787 raft_consensus.cc:482] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:46.187314 23787 raft_consensus.cc:504] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:46.187809 23787 leader_election.cc:288] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:46.188200 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8f2b775fa54c44a5d855d586576696" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242" is_pre_election: true
I20240502 14:15:46.188412 22575 raft_consensus.cc:2454] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:46.188388 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8f2b775fa54c44a5d855d586576696" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:46.188613 22310 raft_consensus.cc:2454] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:46.188827 22642 leader_election.cc:302] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:46.189131 23787 raft_consensus.cc:2781] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:46.189242 23787 raft_consensus.cc:482] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:46.189342 23787 raft_consensus.cc:3037] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:46.191197 23787 raft_consensus.cc:504] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:46.191911 23787 leader_election.cc:288] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:46.192099 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8f2b775fa54c44a5d855d586576696" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:46.192308 22575 raft_consensus.cc:3037] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:46.192261 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8f2b775fa54c44a5d855d586576696" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:46.192435 22310 raft_consensus.cc:3037] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:46.195130 22310 raft_consensus.cc:2454] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:46.195120 22575 raft_consensus.cc:2454] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:46.195477 22642 leader_election.cc:302] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:46.195724 23787 raft_consensus.cc:2781] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:46.195924 23787 raft_consensus.cc:686] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:46.196164 23787 consensus_queue.cc:237] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:46.198746 22170 catalog_manager.cc:5484] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
W20240502 14:15:46.537617 23859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:46.537905 23859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:46.544811 23859 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:15:46.552951 23855 consensus_queue.cc:1035] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:46.558252 23709 consensus_queue.cc:1035] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:46.598968 23855 consensus_queue.cc:1035] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
I20240502 14:15:46.608731 23787 consensus_queue.cc:1035] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:46.653692 23320 consensus_queue.cc:1035] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:46.667246 23320 consensus_queue.cc:1035] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 292.53 ms
time per row: 0.146265 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:15:47.942396 22164 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:47.951807 22423 tablet_service.cc:1467] Processing CreateTablet for tablet eb34b5ac0e8e4c15a1df9a62e439c56c (DEFAULT_TABLE table=rebalance_test_table_7 [id=7c43395ba51b460ea5fcf6beb29c1164]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:47.951814 22555 tablet_service.cc:1467] Processing CreateTablet for tablet eb34b5ac0e8e4c15a1df9a62e439c56c (DEFAULT_TABLE table=rebalance_test_table_7 [id=7c43395ba51b460ea5fcf6beb29c1164]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:47.952219 22555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb34b5ac0e8e4c15a1df9a62e439c56c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.952219 22423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb34b5ac0e8e4c15a1df9a62e439c56c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.952708 22291 tablet_service.cc:1467] Processing CreateTablet for tablet eb34b5ac0e8e4c15a1df9a62e439c56c (DEFAULT_TABLE table=rebalance_test_table_7 [id=7c43395ba51b460ea5fcf6beb29c1164]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:47.953270 22291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb34b5ac0e8e4c15a1df9a62e439c56c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.953560 22292 tablet_service.cc:1467] Processing CreateTablet for tablet 5f79b02ca45b4dea90513712bc57e137 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7c43395ba51b460ea5fcf6beb29c1164]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:47.953912 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f79b02ca45b4dea90513712bc57e137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.957580 22686 tablet_service.cc:1467] Processing CreateTablet for tablet 5f79b02ca45b4dea90513712bc57e137 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7c43395ba51b460ea5fcf6beb29c1164]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:47.957934 22686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f79b02ca45b4dea90513712bc57e137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.958359 22556 tablet_service.cc:1467] Processing CreateTablet for tablet 5f79b02ca45b4dea90513712bc57e137 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7c43395ba51b460ea5fcf6beb29c1164]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:47.958717 22556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f79b02ca45b4dea90513712bc57e137. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.959986 22685 tablet_service.cc:1467] Processing CreateTablet for tablet fc76aec8299347378f28f9ac8a8b5ce8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7c43395ba51b460ea5fcf6beb29c1164]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:47.960314 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc76aec8299347378f28f9ac8a8b5ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.961869 22424 tablet_service.cc:1467] Processing CreateTablet for tablet fc76aec8299347378f28f9ac8a8b5ce8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7c43395ba51b460ea5fcf6beb29c1164]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:47.962214 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc76aec8299347378f28f9ac8a8b5ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.962437 22554 tablet_service.cc:1467] Processing CreateTablet for tablet fc76aec8299347378f28f9ac8a8b5ce8 (DEFAULT_TABLE table=rebalance_test_table_7 [id=7c43395ba51b460ea5fcf6beb29c1164]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:47.962780 22554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc76aec8299347378f28f9ac8a8b5ce8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:47.977445 23924 tablet_bootstrap.cc:492] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:47.978957 23926 tablet_bootstrap.cc:492] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:47.979372 23925 tablet_bootstrap.cc:492] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:47.979423 23924 tablet_bootstrap.cc:654] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:47.980966 23925 tablet_bootstrap.cc:654] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:47.981004 23926 tablet_bootstrap.cc:654] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:47.981631 23923 tablet_bootstrap.cc:492] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:47.982503 23924 tablet_bootstrap.cc:492] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:47.982697 23924 ts_tablet_manager.cc:1367] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240502 14:15:47.983438 23924 raft_consensus.cc:348] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:47.983713 23923 tablet_bootstrap.cc:654] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:47.983803 23924 raft_consensus.cc:374] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:47.983955 23924 raft_consensus.cc:729] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:47.984289 23924 consensus_queue.cc:260] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:47.985149 23924 ts_tablet_manager.cc:1398] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:47.985567 23924 tablet_bootstrap.cc:492] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:47.987406 23924 tablet_bootstrap.cc:654] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:47.992529 23923 tablet_bootstrap.cc:492] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:47.992730 23923 ts_tablet_manager.cc:1367] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240502 14:15:47.993680 23923 raft_consensus.cc:348] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:47.993988 23923 raft_consensus.cc:374] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:47.994139 23923 raft_consensus.cc:729] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:47.994478 23923 consensus_queue.cc:260] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:47.995178 23925 tablet_bootstrap.cc:492] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:47.995180 23923 ts_tablet_manager.cc:1398] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:47.995290 23926 tablet_bootstrap.cc:492] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:47.995354 23925 ts_tablet_manager.cc:1367] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.009s
I20240502 14:15:47.995394 23923 tablet_bootstrap.cc:492] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:47.995442 23926 ts_tablet_manager.cc:1367] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20240502 14:15:47.996109 23926 raft_consensus.cc:348] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:47.996341 23926 raft_consensus.cc:374] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:47.996472 23926 raft_consensus.cc:729] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:47.996769 23923 tablet_bootstrap.cc:654] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:47.996724 23926 consensus_queue.cc:260] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:47.996932 23925 raft_consensus.cc:348] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:47.997246 23925 raft_consensus.cc:374] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:47.997372 23925 raft_consensus.cc:729] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:47.997643 23925 consensus_queue.cc:260] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:47.998064 23926 ts_tablet_manager.cc:1398] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:47.998358 23926 tablet_bootstrap.cc:492] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:47.999996 23926 tablet_bootstrap.cc:654] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:48.000465 23925 ts_tablet_manager.cc:1398] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240502 14:15:48.000799 23925 tablet_bootstrap.cc:492] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:48.002434 23925 tablet_bootstrap.cc:654] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:48.004630 23923 tablet_bootstrap.cc:492] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:48.004863 23923 ts_tablet_manager.cc:1367] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.009s
I20240502 14:15:48.005645 23923 raft_consensus.cc:348] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:48.005903 23923 raft_consensus.cc:374] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:48.006009 23923 raft_consensus.cc:729] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:48.006304 23923 consensus_queue.cc:260] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:48.007076 23924 tablet_bootstrap.cc:492] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:48.007284 23924 ts_tablet_manager.cc:1367] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.007s
I20240502 14:15:48.007719 23923 ts_tablet_manager.cc:1398] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:15:48.008965 23924 raft_consensus.cc:348] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:48.009215 23924 raft_consensus.cc:374] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:48.009361 23924 raft_consensus.cc:729] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:48.010356 23924 consensus_queue.cc:260] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:48.010825 23924 ts_tablet_manager.cc:1398] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240502 14:15:48.011096 23924 tablet_bootstrap.cc:492] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:48.013036 23924 tablet_bootstrap.cc:654] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:48.013146 23925 tablet_bootstrap.cc:492] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:48.013330 23925 ts_tablet_manager.cc:1367] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240502 14:15:48.013367 23926 tablet_bootstrap.cc:492] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:48.013535 23926 ts_tablet_manager.cc:1367] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.008s
I20240502 14:15:48.014066 23925 raft_consensus.cc:348] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:48.014325 23925 raft_consensus.cc:374] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:48.014427 23925 raft_consensus.cc:729] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:48.014683 23925 consensus_queue.cc:260] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:48.014515 23926 raft_consensus.cc:348] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:48.015247 23925 ts_tablet_manager.cc:1398] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:48.015309 23926 raft_consensus.cc:374] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:48.015440 23926 raft_consensus.cc:729] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:48.015708 23926 consensus_queue.cc:260] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:48.016675 23926 ts_tablet_manager.cc:1398] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240502 14:15:48.018765 23924 tablet_bootstrap.cc:492] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:48.018934 23924 ts_tablet_manager.cc:1367] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240502 14:15:48.019440 23924 raft_consensus.cc:348] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:48.019630 23924 raft_consensus.cc:374] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:48.019723 23924 raft_consensus.cc:729] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:48.019924 23924 consensus_queue.cc:260] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:48.020346 23924 ts_tablet_manager.cc:1398] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:48.059532 23890 raft_consensus.cc:482] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:48.059669 23890 raft_consensus.cc:504] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:48.060582 23890 leader_election.cc:288] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:48.061079 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:48.061309 22444 raft_consensus.cc:2454] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 0.
I20240502 14:15:48.061681 22248 leader_election.cc:302] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:48.061902 23890 raft_consensus.cc:2781] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:48.062019 23890 raft_consensus.cc:482] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:48.062110 23890 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:48.064152 23890 raft_consensus.cc:504] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:48.064735 23890 leader_election.cc:288] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:48.065162 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:48.065342 22444 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:48.066999 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242" is_pre_election: true
I20240502 14:15:48.067174 22575 raft_consensus.cc:2454] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 0.
I20240502 14:15:48.067143 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:48.067310 22576 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:48.067703 22444 raft_consensus.cc:2454] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 1.
I20240502 14:15:48.068032 22248 leader_election.cc:302] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:48.068311 23890 raft_consensus.cc:2781] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:48.068636 23890 raft_consensus.cc:686] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:48.068857 23890 consensus_queue.cc:237] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:48.069413 22576 raft_consensus.cc:2454] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 1.
I20240502 14:15:48.071354 22164 catalog_manager.cc:5484] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:48.114272 23780 raft_consensus.cc:482] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:48.114418 23780 raft_consensus.cc:504] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:48.115012 23780 leader_election.cc:288] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:48.115365 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79b02ca45b4dea90513712bc57e137" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:48.115427 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79b02ca45b4dea90513712bc57e137" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:48.115581 22708 raft_consensus.cc:2454] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:48.115593 22312 raft_consensus.cc:2454] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:48.115962 22509 leader_election.cc:302] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:48.116195 23780 raft_consensus.cc:2781] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:48.116279 23780 raft_consensus.cc:482] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:48.116385 23780 raft_consensus.cc:3037] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:48.118173 23780 raft_consensus.cc:504] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:48.118650 23780 leader_election.cc:288] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:48.119005 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79b02ca45b4dea90513712bc57e137" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:48.119148 22708 raft_consensus.cc:3037] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:48.119091 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79b02ca45b4dea90513712bc57e137" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:48.119256 22312 raft_consensus.cc:3037] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:48.120834 22708 raft_consensus.cc:2454] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:48.121002 22312 raft_consensus.cc:2454] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:48.121224 22509 leader_election.cc:302] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:48.121495 23780 raft_consensus.cc:2781] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:48.121892 23780 raft_consensus.cc:686] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:48.122093 23780 consensus_queue.cc:237] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:48.124231 22164 catalog_manager.cc:5484] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:48.225191 23787 raft_consensus.cc:482] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:48.225404 23787 raft_consensus.cc:504] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:48.226025 23787 leader_election.cc:288] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:48.226536 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242" is_pre_election: true
I20240502 14:15:48.226545 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:48.226737 22576 raft_consensus.cc:2454] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:48.226783 22444 raft_consensus.cc:2454] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:48.227114 22642 leader_election.cc:302] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:48.227525 23787 raft_consensus.cc:2781] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:48.227607 23787 raft_consensus.cc:482] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:48.227679 23787 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:48.229697 23787 raft_consensus.cc:504] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:48.230247 23787 leader_election.cc:288] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:48.230577 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:48.230620 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:48.230751 22444 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:48.230794 22576 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:48.232547 22576 raft_consensus.cc:2454] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:48.232920 22642 leader_election.cc:302] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:48.232933 22444 raft_consensus.cc:2454] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:48.233132 23787 raft_consensus.cc:2781] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:48.233296 23787 raft_consensus.cc:686] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:48.233530 23787 consensus_queue.cc:237] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:48.236346 22164 catalog_manager.cc:5484] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } }
W20240502 14:15:48.562969 23944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:48.563220 23944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:48.570680 23944 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:15:48.596644 23890 consensus_queue.cc:1035] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240502 14:15:48.614589 23890 consensus_queue.cc:1035] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:48.639269 23942 consensus_queue.cc:1035] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:48.644270 23787 consensus_queue.cc:1035] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:48.736452 22708 raft_consensus.cc:1261] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240502 14:15:48.737047 23780 consensus_queue.cc:1035] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:48.745585 23780 consensus_queue.cc:1035] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 302.101 ms
time per row: 0.151051 ms
I20240502 14:15:49.048220 22170 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:49.058403 22423 tablet_service.cc:1467] Processing CreateTablet for tablet eff83cace8554f40beaffe89cb1b684d (DEFAULT_TABLE table=rebalance_test_table_8 [id=ecc4022dcb4848e899662b369fa11615]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:49.058917 22423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff83cace8554f40beaffe89cb1b684d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:49.059119 22555 tablet_service.cc:1467] Processing CreateTablet for tablet fa2f5692b87f4ae0983006b09ffc4cbd (DEFAULT_TABLE table=rebalance_test_table_8 [id=ecc4022dcb4848e899662b369fa11615]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:49.059509 22555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa2f5692b87f4ae0983006b09ffc4cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:49.060195 22686 tablet_service.cc:1467] Processing CreateTablet for tablet eff83cace8554f40beaffe89cb1b684d (DEFAULT_TABLE table=rebalance_test_table_8 [id=ecc4022dcb4848e899662b369fa11615]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:49.060364 22554 tablet_service.cc:1467] Processing CreateTablet for tablet b9e35e0f95454e54a3603a7008026f0f (DEFAULT_TABLE table=rebalance_test_table_8 [id=ecc4022dcb4848e899662b369fa11615]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:49.060583 22686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff83cace8554f40beaffe89cb1b684d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:49.060704 22554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9e35e0f95454e54a3603a7008026f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:49.061528 22685 tablet_service.cc:1467] Processing CreateTablet for tablet b9e35e0f95454e54a3603a7008026f0f (DEFAULT_TABLE table=rebalance_test_table_8 [id=ecc4022dcb4848e899662b369fa11615]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:49.061823 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9e35e0f95454e54a3603a7008026f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:49.064251 22292 tablet_service.cc:1467] Processing CreateTablet for tablet eff83cace8554f40beaffe89cb1b684d (DEFAULT_TABLE table=rebalance_test_table_8 [id=ecc4022dcb4848e899662b369fa11615]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:49.064718 22292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eff83cace8554f40beaffe89cb1b684d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:49.069545 22291 tablet_service.cc:1467] Processing CreateTablet for tablet fa2f5692b87f4ae0983006b09ffc4cbd (DEFAULT_TABLE table=rebalance_test_table_8 [id=ecc4022dcb4848e899662b369fa11615]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:49.069854 22291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa2f5692b87f4ae0983006b09ffc4cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:49.073498 22290 tablet_service.cc:1467] Processing CreateTablet for tablet b9e35e0f95454e54a3603a7008026f0f (DEFAULT_TABLE table=rebalance_test_table_8 [id=ecc4022dcb4848e899662b369fa11615]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:49.073853 22290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9e35e0f95454e54a3603a7008026f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:49.080065 24017 tablet_bootstrap.cc:492] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:49.081588 22424 tablet_service.cc:1467] Processing CreateTablet for tablet fa2f5692b87f4ae0983006b09ffc4cbd (DEFAULT_TABLE table=rebalance_test_table_8 [id=ecc4022dcb4848e899662b369fa11615]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:49.082022 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa2f5692b87f4ae0983006b09ffc4cbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:49.082788 24017 tablet_bootstrap.cc:654] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:49.091432 24018 tablet_bootstrap.cc:492] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:49.099360 24020 tablet_bootstrap.cc:492] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:49.101060 24018 tablet_bootstrap.cc:654] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:49.101408 24020 tablet_bootstrap.cc:654] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:49.104136 24016 tablet_bootstrap.cc:492] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:49.105535 24017 tablet_bootstrap.cc:492] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:49.105612 24018 tablet_bootstrap.cc:492] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:49.105749 24017 ts_tablet_manager.cc:1367] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.004s
I20240502 14:15:49.105880 24018 ts_tablet_manager.cc:1367] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20240502 14:15:49.106673 24017 raft_consensus.cc:348] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:49.106940 24017 raft_consensus.cc:374] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:49.106772 24018 raft_consensus.cc:348] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:49.107041 24017 raft_consensus.cc:729] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:49.107214 24018 raft_consensus.cc:374] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:49.107386 24018 raft_consensus.cc:729] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:49.107307 24017 consensus_queue.cc:260] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:49.107924 24017 ts_tablet_manager.cc:1398] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240502 14:15:49.107857 24018 consensus_queue.cc:260] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:49.108278 24017 tablet_bootstrap.cc:492] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:49.109103 24016 tablet_bootstrap.cc:654] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:49.110105 24017 tablet_bootstrap.cc:654] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:49.111996 24018 ts_tablet_manager.cc:1398] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:15:49.112895 24018 tablet_bootstrap.cc:492] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:49.114902 24020 tablet_bootstrap.cc:492] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:49.115049 24020 ts_tablet_manager.cc:1367] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20240502 14:15:49.115912 24020 raft_consensus.cc:348] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:49.116102 24020 raft_consensus.cc:374] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:49.116228 24020 raft_consensus.cc:729] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:49.116447 24020 consensus_queue.cc:260] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:49.117036 24020 ts_tablet_manager.cc:1398] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:49.117205 24018 tablet_bootstrap.cc:654] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:49.117528 24020 tablet_bootstrap.cc:492] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:49.118892 24020 tablet_bootstrap.cc:654] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:49.120291 24016 tablet_bootstrap.cc:492] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:49.120465 24016 ts_tablet_manager.cc:1367] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240502 14:15:49.121387 24016 raft_consensus.cc:348] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:49.122123 24016 raft_consensus.cc:374] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:49.122388 24016 raft_consensus.cc:729] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:49.122766 24016 consensus_queue.cc:260] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:49.125038 24016 ts_tablet_manager.cc:1398] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240502 14:15:49.125749 24016 tablet_bootstrap.cc:492] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:49.129509 24016 tablet_bootstrap.cc:654] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:49.132138 24020 tablet_bootstrap.cc:492] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:49.132339 24020 ts_tablet_manager.cc:1367] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.000s
I20240502 14:15:49.133201 24020 raft_consensus.cc:348] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:49.133474 24020 raft_consensus.cc:374] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:49.133586 24020 raft_consensus.cc:729] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:49.133936 24020 consensus_queue.cc:260] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:49.134618 24020 ts_tablet_manager.cc:1398] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240502 14:15:49.135010 24020 tablet_bootstrap.cc:492] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:49.136408 24020 tablet_bootstrap.cc:654] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:49.143450 24017 tablet_bootstrap.cc:492] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:49.143694 24017 ts_tablet_manager.cc:1367] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.035s user 0.021s sys 0.010s
I20240502 14:15:49.144511 24018 tablet_bootstrap.cc:492] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:49.144600 24017 raft_consensus.cc:348] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:49.144742 24018 ts_tablet_manager.cc:1367] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.007s
I20240502 14:15:49.145046 24017 raft_consensus.cc:374] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:49.145180 24017 raft_consensus.cc:729] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:49.145254 24020 tablet_bootstrap.cc:492] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:49.145412 24020 ts_tablet_manager.cc:1367] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20240502 14:15:49.145601 24018 raft_consensus.cc:348] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:49.145644 24017 consensus_queue.cc:260] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:49.145874 24018 raft_consensus.cc:374] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:49.145967 24018 raft_consensus.cc:729] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:49.146180 24020 raft_consensus.cc:348] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:49.146412 24018 consensus_queue.cc:260] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:49.146533 24017 ts_tablet_manager.cc:1398] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:49.147300 24016 tablet_bootstrap.cc:492] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:49.147461 24016 ts_tablet_manager.cc:1367] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.000s
I20240502 14:15:49.148213 23725 raft_consensus.cc:482] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:49.148366 23725 raft_consensus.cc:504] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:49.148963 23725 leader_election.cc:288] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:49.149171 24016 raft_consensus.cc:348] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:49.149389 24016 raft_consensus.cc:374] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:49.149457 24016 raft_consensus.cc:729] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:49.149610 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:49.149703 24016 consensus_queue.cc:260] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:49.149847 22312 raft_consensus.cc:2454] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 0.
I20240502 14:15:49.150256 24018 ts_tablet_manager.cc:1398] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240502 14:15:49.150404 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:49.150583 22707 raft_consensus.cc:2454] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 0.
I20240502 14:15:49.150892 24016 ts_tablet_manager.cc:1398] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240502 14:15:49.146364 24020 raft_consensus.cc:374] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:49.151118 24020 raft_consensus.cc:729] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:49.151245 22377 leader_election.cc:302] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:49.151512 23725 raft_consensus.cc:2781] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:49.151607 23725 raft_consensus.cc:482] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:49.151715 23725 raft_consensus.cc:3037] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:49.153796 23725 raft_consensus.cc:504] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:49.154317 23725 leader_election.cc:288] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:49.155092 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:49.151343 24020 consensus_queue.cc:260] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:49.155315 22707 raft_consensus.cc:3037] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:49.155659 24020 ts_tablet_manager.cc:1398] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240502 14:15:49.157585 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:49.157779 22312 raft_consensus.cc:3037] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:49.159559 22312 raft_consensus.cc:2454] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 1.
I20240502 14:15:49.159565 22707 raft_consensus.cc:2454] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 1.
I20240502 14:15:49.159900 22377 leader_election.cc:302] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:49.160159 23725 raft_consensus.cc:2781] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:49.162787 23725 raft_consensus.cc:686] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 1 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:49.163057 23725 consensus_queue.cc:237] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:49.165812 22165 catalog_manager.cc:5484] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 1 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:49.175805 23958 raft_consensus.cc:482] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:49.175908 23958 raft_consensus.cc:504] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:49.176470 23958 leader_election.cc:288] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:49.176854 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242" is_pre_election: true
I20240502 14:15:49.176955 22442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:49.177022 22576 raft_consensus.cc:2454] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 0.
I20240502 14:15:49.177170 22442 raft_consensus.cc:2454] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 0.
I20240502 14:15:49.177461 22247 leader_election.cc:302] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:49.177692 23958 raft_consensus.cc:2781] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:49.177793 23958 raft_consensus.cc:482] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:49.177870 23958 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:49.179374 23958 raft_consensus.cc:504] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:49.180016 23958 leader_election.cc:288] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:49.180970 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:49.180968 22442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:49.181152 22442 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:49.181149 22576 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:49.183681 22442 raft_consensus.cc:2454] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 1.
I20240502 14:15:49.183681 22576 raft_consensus.cc:2454] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 1.
I20240502 14:15:49.184121 22248 leader_election.cc:302] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:49.184455 23958 raft_consensus.cc:2781] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:49.184607 23958 raft_consensus.cc:686] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:49.184825 23958 consensus_queue.cc:237] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:49.187083 22165 catalog_manager.cc:5484] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:49.255785 23958 raft_consensus.cc:482] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:49.255995 23958 raft_consensus.cc:504] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:49.256539 23958 leader_election.cc:288] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:49.256829 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:49.257014 22707 raft_consensus.cc:2454] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 0.
I20240502 14:15:49.257027 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242" is_pre_election: true
I20240502 14:15:49.257203 22576 raft_consensus.cc:2454] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 0.
I20240502 14:15:49.257401 22245 leader_election.cc:302] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:49.257668 23958 raft_consensus.cc:2781] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:49.257772 23958 raft_consensus.cc:482] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:49.257877 23958 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:49.260777 23958 raft_consensus.cc:504] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:49.261512 23958 leader_election.cc:288] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:49.261812 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:49.261816 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:49.261960 22576 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:49.262020 22707 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:49.263447 22576 raft_consensus.cc:2454] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 1.
I20240502 14:15:49.263738 22247 leader_election.cc:302] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:49.263983 23958 raft_consensus.cc:2781] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:49.264148 23958 raft_consensus.cc:686] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:49.264396 23958 consensus_queue.cc:237] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:49.265216 22707 raft_consensus.cc:2454] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 1.
I20240502 14:15:49.267050 22165 catalog_manager.cc:5484] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 1 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } }
W20240502 14:15:49.591053 24033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:49.591303 24033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:49.598033 24033 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240502 14:15:49.617734 23890 consensus_queue.cc:1035] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:49.625615 23890 consensus_queue.cc:1035] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
I20240502 14:15:49.707561 23890 consensus_queue.cc:1035] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:49.724501 23725 consensus_queue.cc:1035] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:49.728422 23725 consensus_queue.cc:1035] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:49.776335 24046 consensus_queue.cc:1035] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 252.829 ms
time per row: 0.126415 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:15:50.860682 22165 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35604:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240502 14:15:50.869266 22423 tablet_service.cc:1467] Processing CreateTablet for tablet 5b08f8d408bd42eb8f1f9ca80fe176c1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=58a9485c7dc74789994c32a5f31dee45]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:50.869693 22423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b08f8d408bd42eb8f1f9ca80fe176c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:50.870554 22422 tablet_service.cc:1467] Processing CreateTablet for tablet a2c0dc3c5bd24a8ebea5f15f037e7a2a (DEFAULT_TABLE table=rebalance_test_table_9 [id=58a9485c7dc74789994c32a5f31dee45]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:50.870913 22422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c0dc3c5bd24a8ebea5f15f037e7a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:50.871330 22554 tablet_service.cc:1467] Processing CreateTablet for tablet 5b08f8d408bd42eb8f1f9ca80fe176c1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=58a9485c7dc74789994c32a5f31dee45]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:50.871697 22554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b08f8d408bd42eb8f1f9ca80fe176c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:50.871623 22291 tablet_service.cc:1467] Processing CreateTablet for tablet a2c0dc3c5bd24a8ebea5f15f037e7a2a (DEFAULT_TABLE table=rebalance_test_table_9 [id=58a9485c7dc74789994c32a5f31dee45]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:50.872014 22291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c0dc3c5bd24a8ebea5f15f037e7a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:50.873628 22424 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf0ec2ecc7d455db867747073aaea0e (DEFAULT_TABLE table=rebalance_test_table_9 [id=58a9485c7dc74789994c32a5f31dee45]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:50.874064 22424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf0ec2ecc7d455db867747073aaea0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:50.876091 22555 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf0ec2ecc7d455db867747073aaea0e (DEFAULT_TABLE table=rebalance_test_table_9 [id=58a9485c7dc74789994c32a5f31dee45]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:50.876410 22555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf0ec2ecc7d455db867747073aaea0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:50.877635 22685 tablet_service.cc:1467] Processing CreateTablet for tablet 5b08f8d408bd42eb8f1f9ca80fe176c1 (DEFAULT_TABLE table=rebalance_test_table_9 [id=58a9485c7dc74789994c32a5f31dee45]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:50.878039 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b08f8d408bd42eb8f1f9ca80fe176c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:50.881676 22686 tablet_service.cc:1467] Processing CreateTablet for tablet 5bf0ec2ecc7d455db867747073aaea0e (DEFAULT_TABLE table=rebalance_test_table_9 [id=58a9485c7dc74789994c32a5f31dee45]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:50.882136 22686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bf0ec2ecc7d455db867747073aaea0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:50.885279 24101 tablet_bootstrap.cc:492] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6: Bootstrap starting.
I20240502 14:15:50.885568 22688 tablet_service.cc:1467] Processing CreateTablet for tablet a2c0dc3c5bd24a8ebea5f15f037e7a2a (DEFAULT_TABLE table=rebalance_test_table_9 [id=58a9485c7dc74789994c32a5f31dee45]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240502 14:15:50.885876 22688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2c0dc3c5bd24a8ebea5f15f037e7a2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240502 14:15:50.887785 24101 tablet_bootstrap.cc:654] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:50.891602 24102 tablet_bootstrap.cc:492] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:50.895150 24102 tablet_bootstrap.cc:654] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:50.904314 24102 tablet_bootstrap.cc:492] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:50.904568 24102 ts_tablet_manager.cc:1367] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.006s
I20240502 14:15:50.905866 24102 raft_consensus.cc:348] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:50.906227 24102 raft_consensus.cc:374] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:50.906366 24102 raft_consensus.cc:729] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:50.906677 24102 consensus_queue.cc:260] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:50.907604 24105 tablet_bootstrap.cc:492] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:50.909114 24105 tablet_bootstrap.cc:654] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:50.910945 24100 tablet_bootstrap.cc:492] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:50.911262 24101 tablet_bootstrap.cc:492] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6: No bootstrap required, opened a new log
I20240502 14:15:50.911476 24101 ts_tablet_manager.cc:1367] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.006s
I20240502 14:15:50.911976 24102 ts_tablet_manager.cc:1398] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240502 14:15:50.912242 24102 tablet_bootstrap.cc:492] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242: Bootstrap starting.
I20240502 14:15:50.912279 24101 raft_consensus.cc:348] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:50.912546 24100 tablet_bootstrap.cc:654] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:50.912572 24101 raft_consensus.cc:374] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:50.912751 24101 raft_consensus.cc:729] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Initialized, Role: FOLLOWER
I20240502 14:15:50.913060 24101 consensus_queue.cc:260] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:50.914213 24105 tablet_bootstrap.cc:492] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:50.914366 24105 ts_tablet_manager.cc:1367] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240502 14:15:50.914400 24101 ts_tablet_manager.cc:1398] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240502 14:15:50.915118 24105 raft_consensus.cc:348] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:50.915374 24105 raft_consensus.cc:374] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:50.915511 24105 raft_consensus.cc:729] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:50.915556 24102 tablet_bootstrap.cc:654] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:50.915755 24105 consensus_queue.cc:260] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:50.916321 24105 ts_tablet_manager.cc:1398] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240502 14:15:50.916775 24105 tablet_bootstrap.cc:492] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:50.918409 24105 tablet_bootstrap.cc:654] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:50.919871 24100 tablet_bootstrap.cc:492] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:50.920068 24100 ts_tablet_manager.cc:1367] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20240502 14:15:50.920732 24100 raft_consensus.cc:348] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:50.920970 24100 raft_consensus.cc:374] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:50.921085 24100 raft_consensus.cc:729] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:50.921339 24100 consensus_queue.cc:260] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:50.922478 24100 ts_tablet_manager.cc:1398] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240502 14:15:50.922739 24100 tablet_bootstrap.cc:492] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:50.924355 24100 tablet_bootstrap.cc:654] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:50.928382 24105 tablet_bootstrap.cc:492] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:50.928563 24105 ts_tablet_manager.cc:1367] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240502 14:15:50.928963 24102 tablet_bootstrap.cc:492] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242: No bootstrap required, opened a new log
I20240502 14:15:50.929059 24100 tablet_bootstrap.cc:492] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:50.929111 24102 ts_tablet_manager.cc:1367] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242: Time spent bootstrapping tablet: real 0.017s user 0.013s sys 0.000s
I20240502 14:15:50.929204 24100 ts_tablet_manager.cc:1367] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240502 14:15:50.929307 24105 raft_consensus.cc:348] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:50.929486 24105 raft_consensus.cc:374] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:50.929560 24105 raft_consensus.cc:729] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:50.929754 24100 raft_consensus.cc:348] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:50.929752 24105 consensus_queue.cc:260] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:50.929790 24102 raft_consensus.cc:348] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:50.929900 24100 raft_consensus.cc:374] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:50.929961 24102 raft_consensus.cc:374] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:50.929991 24100 raft_consensus.cc:729] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:50.930035 24102 raft_consensus.cc:729] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Initialized, Role: FOLLOWER
I20240502 14:15:50.930212 24100 consensus_queue.cc:260] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:50.930280 24102 consensus_queue.cc:260] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:50.930914 24102 ts_tablet_manager.cc:1398] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240502 14:15:50.930644 24100 ts_tablet_manager.cc:1398] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240502 14:15:50.931207 24100 tablet_bootstrap.cc:492] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5: Bootstrap starting.
I20240502 14:15:50.932554 24100 tablet_bootstrap.cc:654] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:50.933259 24105 ts_tablet_manager.cc:1398] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240502 14:15:50.933555 24105 tablet_bootstrap.cc:492] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10: Bootstrap starting.
I20240502 14:15:50.935700 24105 tablet_bootstrap.cc:654] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10: Neither blocks nor log segments found. Creating new log.
I20240502 14:15:50.937294 24100 tablet_bootstrap.cc:492] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5: No bootstrap required, opened a new log
I20240502 14:15:50.937438 24100 ts_tablet_manager.cc:1367] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.006s
I20240502 14:15:50.938094 24100 raft_consensus.cc:348] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:50.938242 24100 raft_consensus.cc:374] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:50.938345 24100 raft_consensus.cc:729] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Initialized, Role: FOLLOWER
I20240502 14:15:50.938549 24100 consensus_queue.cc:260] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:50.939016 24100 ts_tablet_manager.cc:1398] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:50.942057 24105 tablet_bootstrap.cc:492] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10: No bootstrap required, opened a new log
I20240502 14:15:50.942183 24105 ts_tablet_manager.cc:1367] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240502 14:15:50.942864 24105 raft_consensus.cc:348] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:50.943027 24105 raft_consensus.cc:374] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240502 14:15:50.943112 24105 raft_consensus.cc:729] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Initialized, Role: FOLLOWER
I20240502 14:15:50.943265 24105 consensus_queue.cc:260] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:50.943701 24105 ts_tablet_manager.cc:1398] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240502 14:15:50.962729 24009 raft_consensus.cc:482] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:50.962872 24009 raft_consensus.cc:504] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:50.963397 24009 leader_election.cc:288] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:50.963702 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:50.963768 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:50.963929 22707 raft_consensus.cc:2454] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:50.964016 22444 raft_consensus.cc:2454] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 0.
I20240502 14:15:50.964285 22509 leader_election.cc:302] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:50.964553 24009 raft_consensus.cc:2781] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:50.964665 24009 raft_consensus.cc:482] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:50.964751 24009 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:50.966727 24009 raft_consensus.cc:504] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:50.967226 24009 leader_election.cc:288] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:50.967554 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:50.967648 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:50.967742 22707 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:50.967804 22444 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:50.970305 22707 raft_consensus.cc:2454] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:50.970362 22444 raft_consensus.cc:2454] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 1.
I20240502 14:15:50.970726 22509 leader_election.cc:302] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:50.971210 24009 raft_consensus.cc:2781] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:50.971374 24009 raft_consensus.cc:686] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:50.971648 24009 consensus_queue.cc:237] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:50.974177 22166 catalog_manager.cc:5484] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 0 to 1, leader changed from <none> to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 1 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:51.034421 23999 raft_consensus.cc:482] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:51.034586 23999 raft_consensus.cc:504] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:51.035255 23999 leader_election.cc:288] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:51.035564 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:51.035813 22444 raft_consensus.cc:2454] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:51.035748 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242" is_pre_election: true
I20240502 14:15:51.035979 22575 raft_consensus.cc:2454] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:51.036190 22643 leader_election.cc:302] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:51.036449 23999 raft_consensus.cc:2781] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:51.036562 23999 raft_consensus.cc:482] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:51.036650 23999 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:51.038684 23999 raft_consensus.cc:504] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:51.039283 23999 leader_election.cc:288] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:51.039561 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:51.039768 22444 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:51.039708 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:51.039898 22575 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:51.042618 22444 raft_consensus.cc:2454] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:51.042631 22575 raft_consensus.cc:2454] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:51.042999 22643 leader_election.cc:302] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:51.043285 23999 raft_consensus.cc:2781] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:51.043634 23999 raft_consensus.cc:686] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:51.043926 23999 consensus_queue.cc:237] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:51.046754 22166 catalog_manager.cc:5484] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:51.137006 23999 raft_consensus.cc:482] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:51.137212 23999 raft_consensus.cc:504] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:51.137753 23999 leader_election.cc:288] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:51.138167 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5" is_pre_election: true
I20240502 14:15:51.138217 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:51.138353 22444 raft_consensus.cc:2454] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:51.138409 22312 raft_consensus.cc:2454] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 0.
I20240502 14:15:51.138726 22643 leader_election.cc:302] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:51.138998 23999 raft_consensus.cc:2781] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240502 14:15:51.139086 23999 raft_consensus.cc:482] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:51.139176 23999 raft_consensus.cc:3037] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:51.141326 23999 raft_consensus.cc:504] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:51.141933 23999 leader_election.cc:288] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:51.142320 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:51.142331 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:51.142467 22444 raft_consensus.cc:3037] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:51.142469 22312 raft_consensus.cc:3037] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 0 FOLLOWER]: Advancing to term 1
I20240502 14:15:51.144093 22444 raft_consensus.cc:2454] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:51.144112 22312 raft_consensus.cc:2454] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 1.
I20240502 14:15:51.144456 22643 leader_election.cc:302] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:51.144692 23999 raft_consensus.cc:2781] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Leader election won for term 1
I20240502 14:15:51.144852 23999 raft_consensus.cc:686] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:51.145104 23999 consensus_queue.cc:237] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:51.147912 22166 catalog_manager.cc:5484] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 0 to 1, leader changed from <none> to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 1 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:51.465279 24113 consensus_queue.cc:1035] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:51.471213 24121 consensus_queue.cc:1035] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20240502 14:15:51.500121 24118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:51.500381 24118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:51.507288 24118 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240502 14:15:51.545763 23999 consensus_queue.cc:1035] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:51.549839 24115 consensus_queue.cc:1035] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:51.585186 23999 consensus_queue.cc:1035] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240502 14:15:51.592801 24115 consensus_queue.cc:1035] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240502 14:15:51.752684 22642 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.196.195:42203, user_credentials={real_user=slave}} blocked reactor thread for 51927.1us
INSERT report
rows total: 2000
time total: 237.193 ms
time per row: 0.118596 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240502 14:15:52.764093 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.197:42099
--local_ip_for_outbound_sockets=127.6.196.197
--tserver_master_addrs=127.6.196.254:33767
--webserver_port=35295
--webserver_interface=127.6.196.197
--builtin_ntp_servers=127.6.196.212:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240502 14:15:53.065868 24182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:53.066130 24182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:53.066258 24182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:53.073535 24182 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:53.073655 24182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:53.073788 24182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.197
I20240502 14:15:53.082259 24182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.197:42099
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.6.196.197
--webserver_port=35295
--tserver_master_addrs=127.6.196.254:33767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.197
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:53.083406 24182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:53.085554 24182 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:53.097399 24187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:53.098136 24188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:53.098671 24190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:53.099411 24182 server_base.cc:1034] running on GCE node
I20240502 14:15:53.100062 24182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:53.101430 24182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:53.102631 24182 hybrid_clock.cc:648] HybridClock initialized: now 1714659353102595 us; error 32 us; skew 500 ppm
I20240502 14:15:53.102941 24182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:53.105158 24182 webserver.cc:460] Webserver started at http://127.6.196.197:35295/ using document root <none> and password file <none>
I20240502 14:15:53.106002 24182 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:53.106128 24182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:53.111354 24182 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20240502 14:15:53.113674 24197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.114933 24182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20240502 14:15:53.115085 24182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "06f023ab0eed430088cc1ce6a17d8418"
format_stamp: "Formatted at 2024-05-02 14:15:31 on dist-test-slave-kjf7"
I20240502 14:15:53.115794 24182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:53.128372 24182 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:53.129174 24182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:53.129462 24182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:53.130313 24182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:53.131615 24182 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:53.131683 24182 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.131809 24182 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:53.131860 24182 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.161535 24182 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.197:42099
I20240502 14:15:53.161549 24310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.197:42099 every 8 connection(s)
I20240502 14:15:53.162794 24182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240502 14:15:53.163702 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 24182
I20240502 14:15:53.164443 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.198:37151
--local_ip_for_outbound_sockets=127.6.196.198
--tserver_master_addrs=127.6.196.254:33767
--webserver_port=33623
--webserver_interface=127.6.196.198
--builtin_ntp_servers=127.6.196.212:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:53.175475 24311 heartbeater.cc:347] Connected to a master server at 127.6.196.254:33767
I20240502 14:15:53.175724 24311 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:53.176524 24311 heartbeater.cc:511] Master 127.6.196.254:33767 requested a full tablet report, sending...
I20240502 14:15:53.178355 22166 ts_manager.cc:194] Re-registered known tserver with Master: 06f023ab0eed430088cc1ce6a17d8418 (127.6.196.197:42099)
I20240502 14:15:53.179291 22166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.197:38509
W20240502 14:15:53.479858 24315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:53.480187 24315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:53.480330 24315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:53.487560 24315 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:53.487694 24315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:53.487848 24315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.198
I20240502 14:15:53.496196 24315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.198:37151
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.6.196.198
--webserver_port=33623
--tserver_master_addrs=127.6.196.254:33767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.198
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:53.497418 24315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:53.499001 24315 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:53.511062 24321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:53.511763 24322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:53.512384 24324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:53.513175 24315 server_base.cc:1034] running on GCE node
I20240502 14:15:53.513695 24315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:53.514516 24315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:53.515699 24315 hybrid_clock.cc:648] HybridClock initialized: now 1714659353515661 us; error 38 us; skew 500 ppm
I20240502 14:15:53.516016 24315 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:53.518067 24315 webserver.cc:460] Webserver started at http://127.6.196.198:33623/ using document root <none> and password file <none>
I20240502 14:15:53.518677 24315 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:53.518790 24315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:53.523511 24315 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:15:53.526119 24330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.527375 24315 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240502 14:15:53.527532 24315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "d50d05f10b0c49528cfacfefe923edf2"
format_stamp: "Formatted at 2024-05-02 14:15:32 on dist-test-slave-kjf7"
I20240502 14:15:53.528326 24315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:53.557873 24315 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:53.558570 24315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:53.558869 24315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:53.559782 24315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:53.561323 24315 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:53.561396 24315 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.561486 24315 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:53.561532 24315 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.592955 24315 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.198:37151
I20240502 14:15:53.593001 24443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.198:37151 every 8 connection(s)
I20240502 14:15:53.594441 24315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240502 14:15:53.596735 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 24315
I20240502 14:15:53.597476 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.199:43327
--local_ip_for_outbound_sockets=127.6.196.199
--tserver_master_addrs=127.6.196.254:33767
--webserver_port=33121
--webserver_interface=127.6.196.199
--builtin_ntp_servers=127.6.196.212:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:53.633348 24444 heartbeater.cc:347] Connected to a master server at 127.6.196.254:33767
I20240502 14:15:53.633663 24444 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:53.634500 24444 heartbeater.cc:511] Master 127.6.196.254:33767 requested a full tablet report, sending...
I20240502 14:15:53.635885 22166 ts_manager.cc:194] Re-registered known tserver with Master: d50d05f10b0c49528cfacfefe923edf2 (127.6.196.198:37151)
I20240502 14:15:53.637054 22166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.198:34305
W20240502 14:15:53.913614 24446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:53.913837 24446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:53.913951 24446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:53.921537 24446 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:53.921669 24446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:53.921829 24446 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.199
I20240502 14:15:53.930743 24446 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.199:43327
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.6.196.199
--webserver_port=33121
--tserver_master_addrs=127.6.196.254:33767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.199
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:53.931746 24446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:53.933369 24446 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:53.944442 24456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:53.945374 24454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:53.945885 24453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:53.946624 24446 server_base.cc:1034] running on GCE node
I20240502 14:15:53.947381 24446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:53.948283 24446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:53.949470 24446 hybrid_clock.cc:648] HybridClock initialized: now 1714659353949432 us; error 44 us; skew 500 ppm
I20240502 14:15:53.949769 24446 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:53.956797 24446 webserver.cc:460] Webserver started at http://127.6.196.199:33121/ using document root <none> and password file <none>
I20240502 14:15:53.957490 24446 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:53.957612 24446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:53.962366 24446 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20240502 14:15:53.964905 24462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.966169 24446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:15:53.966310 24446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "e0248ccaeff74f9ea81e0d6181b652ba"
format_stamp: "Formatted at 2024-05-02 14:15:32 on dist-test-slave-kjf7"
I20240502 14:15:53.966981 24446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:53.981629 24446 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:53.982411 24446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:53.982684 24446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:53.983500 24446 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:53.984794 24446 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:53.984864 24446 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:53.984972 24446 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:53.985019 24446 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:54.019088 24446 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.199:43327
I20240502 14:15:54.019120 24577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.199:43327 every 8 connection(s)
I20240502 14:15:54.020574 24446 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240502 14:15:54.022634 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 24446
I20240502 14:15:54.023203 6931 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
/tmp/dist-test-taskDtn6qM/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.196.200:35297
--local_ip_for_outbound_sockets=127.6.196.200
--tserver_master_addrs=127.6.196.254:33767
--webserver_port=39901
--webserver_interface=127.6.196.200
--builtin_ntp_servers=127.6.196.212:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240502 14:15:54.034004 24578 heartbeater.cc:347] Connected to a master server at 127.6.196.254:33767
I20240502 14:15:54.034276 24578 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:54.035005 24578 heartbeater.cc:511] Master 127.6.196.254:33767 requested a full tablet report, sending...
I20240502 14:15:54.036257 22166 ts_manager.cc:194] Re-registered known tserver with Master: e0248ccaeff74f9ea81e0d6181b652ba (127.6.196.199:43327)
I20240502 14:15:54.036948 22166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.199:60935
I20240502 14:15:54.181831 24311 heartbeater.cc:503] Master 127.6.196.254:33767 was elected leader, sending a full tablet report...
W20240502 14:15:54.344679 24581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240502 14:15:54.344944 24581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240502 14:15:54.345101 24581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240502 14:15:54.352099 24581 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240502 14:15:54.352213 24581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240502 14:15:54.352319 24581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.196.200
I20240502 14:15:54.360769 24581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.196.212:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.196.200:35297
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.6.196.200
--webserver_port=39901
--tserver_master_addrs=127.6.196.254:33767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.196.200
--log_dir=/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision a6923a82b532b3d5c86547dfd9b0772bc97ae884
build type FASTDEBUG
built by None at 02 May 2024 13:43:20 UTC on efd1337a2aa3
build id 91
ASAN enabled
I20240502 14:15:54.361867 24581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240502 14:15:54.363448 24581 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240502 14:15:54.375941 24589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:54.376466 24587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240502 14:15:54.376942 24586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240502 14:15:54.377728 24581 server_base.cc:1034] running on GCE node
I20240502 14:15:54.378305 24581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240502 14:15:54.379144 24581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240502 14:15:54.380337 24581 hybrid_clock.cc:648] HybridClock initialized: now 1714659354380296 us; error 32 us; skew 500 ppm
I20240502 14:15:54.380645 24581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240502 14:15:54.388021 24581 webserver.cc:460] Webserver started at http://127.6.196.200:39901/ using document root <none> and password file <none>
I20240502 14:15:54.388676 24581 fs_manager.cc:359] Metadata directory not provided
I20240502 14:15:54.388826 24581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240502 14:15:54.393955 24581 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20240502 14:15:54.396991 24595 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:54.398393 24581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240502 14:15:54.398525 24581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "7eeefbc07b7c48e09fa850ee106bd0ca"
format_stamp: "Formatted at 2024-05-02 14:15:33 on dist-test-slave-kjf7"
I20240502 14:15:54.399354 24581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240502 14:15:54.427434 24581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240502 14:15:54.428186 24581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240502 14:15:54.428450 24581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240502 14:15:54.429301 24581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240502 14:15:54.430675 24581 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240502 14:15:54.430755 24581 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:54.430891 24581 ts_tablet_manager.cc:590] Registered 0 tablets
I20240502 14:15:54.430949 24581 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240502 14:15:54.471109 24581 rpc_server.cc:307] RPC server started. Bound to: 127.6.196.200:35297
I20240502 14:15:54.471138 24707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.196.200:35297 every 8 connection(s)
I20240502 14:15:54.472461 24581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskDtn6qM/test-tmp/rebalancer_tool-test.1.RebalancingDuringElectionStormTest.RoundRobin_1.1714659208954437-6931-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240502 14:15:54.477363 6931 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskDtn6qM/build/asan/bin/kudu as pid 24581
I20240502 14:15:54.486071 24708 heartbeater.cc:347] Connected to a master server at 127.6.196.254:33767
I20240502 14:15:54.486318 24708 heartbeater.cc:464] Registering TS with master...
I20240502 14:15:54.487083 24708 heartbeater.cc:511] Master 127.6.196.254:33767 requested a full tablet report, sending...
I20240502 14:15:54.488456 22166 ts_manager.cc:194] Re-registered known tserver with Master: 7eeefbc07b7c48e09fa850ee106bd0ca (127.6.196.200:35297)
I20240502 14:15:54.489406 22166 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.196.200:43737
I20240502 14:15:54.517541 22442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.517781 22442 raft_consensus.cc:482] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.517897 22442 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.519668 22442 raft_consensus.cc:504] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.520291 22442 leader_election.cc:288] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:54.520639 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.520841 22576 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.520895 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:54.521093 22707 raft_consensus.cc:3032] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.521186 22707 raft_consensus.cc:729] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.521325 22442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.521458 22442 raft_consensus.cc:482] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.521549 22442 raft_consensus.cc:3037] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.521462 22707 consensus_queue.cc:260] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.521930 22707 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.523288 22576 raft_consensus.cc:2454] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.523710 22379 leader_election.cc:302] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:54.523667 22442 raft_consensus.cc:504] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.523952 24078 raft_consensus.cc:2781] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.524153 22707 raft_consensus.cc:2454] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.524634 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.524922 22312 raft_consensus.cc:3037] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.524984 24078 raft_consensus.cc:686] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.525347 24078 consensus_queue.cc:237] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.526407 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:54.526597 22707 raft_consensus.cc:3032] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.526715 22707 raft_consensus.cc:729] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.526964 22707 consensus_queue.cc:260] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.527278 22707 raft_consensus.cc:3037] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.527832 22312 raft_consensus.cc:2454] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.528401 22377 leader_election.cc:302] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:54.529439 22707 raft_consensus.cc:2454] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.529677 24078 raft_consensus.cc:2781] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.529865 24078 raft_consensus.cc:686] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.529891 22442 leader_election.cc:288] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:54.530205 24078 consensus_queue.cc:237] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.531037 22166 catalog_manager.cc:5484] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.533253 22442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.533469 22442 raft_consensus.cc:465] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 14:15:54.534437 22442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.534687 22442 raft_consensus.cc:482] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.534818 22442 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.538225 22442 raft_consensus.cc:504] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.539494 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:54.539710 22707 raft_consensus.cc:3032] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.539834 22707 raft_consensus.cc:729] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.540160 22707 consensus_queue.cc:260] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.540501 22707 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.542881 22707 raft_consensus.cc:2454] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.543249 22377 leader_election.cc:302] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.543506 24078 raft_consensus.cc:2781] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.543632 24078 raft_consensus.cc:686] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.543677 22442 leader_election.cc:288] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:54.543886 24078 consensus_queue.cc:237] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.544297 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.544438 22444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.544515 22576 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.544654 22444 raft_consensus.cc:482] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.544765 22444 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.547236 22576 raft_consensus.cc:2454] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.547343 22444 raft_consensus.cc:504] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.547953 22444 leader_election.cc:288] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:54.547719 22166 catalog_manager.cc:5484] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.548305 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.548501 22312 raft_consensus.cc:3032] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.548596 22312 raft_consensus.cc:729] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:54.548391 22166 catalog_manager.cc:5484] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.548871 22444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.549022 22444 raft_consensus.cc:482] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.548933 22312 consensus_queue.cc:260] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.549118 22444 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.549551 22312 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.551270 22444 raft_consensus.cc:504] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.551781 22444 leader_election.cc:288] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:54.551908 22312 raft_consensus.cc:2454] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.552163 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.552367 22310 raft_consensus.cc:3032] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.552445 22310 raft_consensus.cc:729] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:54.552556 22444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.552703 22444 raft_consensus.cc:482] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.552680 22310 consensus_queue.cc:260] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.552806 22444 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.552945 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.553009 22310 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.553138 22575 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.555320 22444 raft_consensus.cc:504] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.555583 22575 raft_consensus.cc:2454] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.555881 22377 leader_election.cc:302] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:54.555886 22444 leader_election.cc:288] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:54.556308 24078 raft_consensus.cc:2781] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.556468 24078 raft_consensus.cc:686] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.556690 24078 consensus_queue.cc:237] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.556900 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:54.557055 22707 raft_consensus.cc:3032] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.557157 22707 raft_consensus.cc:729] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.557368 22707 consensus_queue.cc:260] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.557477 22442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.557612 22442 raft_consensus.cc:482] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.557719 22442 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.557726 22707 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.557986 22379 leader_election.cc:302] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:54.558264 24718 raft_consensus.cc:2781] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.560016 22707 raft_consensus.cc:2454] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.560083 22442 raft_consensus.cc:504] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.560922 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.561110 22311 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.561687 22310 raft_consensus.cc:2454] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.561983 22377 leader_election.cc:302] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.562505 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.562660 22576 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.563882 22442 leader_election.cc:288] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:54.563983 22311 raft_consensus.cc:2454] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.564325 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.564481 22575 raft_consensus.cc:3032] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.564574 22575 raft_consensus.cc:729] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:54.565209 22575 consensus_queue.cc:260] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.565363 22442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.565711 22442 raft_consensus.cc:482] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.565747 22575 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.565953 22442 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.566200 24728 raft_consensus.cc:2781] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.566350 24728 raft_consensus.cc:686] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.566430 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:54.566601 22707 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.566676 24728 consensus_queue.cc:237] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.569012 22442 raft_consensus.cc:504] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.570022 22707 raft_consensus.cc:2454] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.570852 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.571139 22312 raft_consensus.cc:3032] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.571285 22312 raft_consensus.cc:729] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:54.571504 22377 leader_election.cc:302] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.571597 22575 raft_consensus.cc:2454] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.571780 24727 raft_consensus.cc:2781] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.571913 22312 consensus_queue.cc:260] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.572355 22312 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.573146 22576 raft_consensus.cc:2454] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.573576 24718 raft_consensus.cc:686] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.573995 24718 consensus_queue.cc:237] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.575927 22312 raft_consensus.cc:2454] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.576381 22377 leader_election.cc:302] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:54.576632 24078 raft_consensus.cc:2781] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.576759 24078 raft_consensus.cc:686] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.577001 24078 consensus_queue.cc:237] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.578249 22442 leader_election.cc:288] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:54.579178 22444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.579205 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:54.579391 22707 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.579401 22444 raft_consensus.cc:482] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.579535 22444 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.582175 22444 raft_consensus.cc:504] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.582698 22707 raft_consensus.cc:2454] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.582753 22444 leader_election.cc:288] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:54.583843 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:54.583878 22442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.584039 22442 raft_consensus.cc:482] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.584035 22707 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.584148 22442 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.584090 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.584281 22312 raft_consensus.cc:3032] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.584372 22312 raft_consensus.cc:729] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:54.584604 22312 consensus_queue.cc:260] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.584978 22312 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.586584 22707 raft_consensus.cc:2454] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.586791 22442 raft_consensus.cc:504] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.587280 22312 raft_consensus.cc:2454] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.587383 22442 leader_election.cc:288] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:54.587837 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.587916 22377 leader_election.cc:302] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.588028 22575 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.588341 22442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.588531 22442 raft_consensus.cc:482] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.588620 22442 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.588776 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:54.588938 22707 raft_consensus.cc:3032] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.589040 22707 raft_consensus.cc:729] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.589354 22707 consensus_queue.cc:260] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.589679 22707 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.590715 22575 raft_consensus.cc:2454] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.591190 22379 leader_election.cc:302] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:54.591215 22442 raft_consensus.cc:504] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.591491 24078 raft_consensus.cc:2781] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.591629 24078 raft_consensus.cc:686] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.591828 24078 consensus_queue.cc:237] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.592063 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.592229 22575 raft_consensus.cc:3032] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.592293 22575 raft_consensus.cc:729] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:54.592571 22575 consensus_queue.cc:260] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.593161 22575 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.595784 22575 raft_consensus.cc:2454] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.596316 22379 leader_election.cc:302] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:54.596632 24718 raft_consensus.cc:2781] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.598438 24730 raft_consensus.cc:2781] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.598881 24730 raft_consensus.cc:686] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.599378 24730 consensus_queue.cc:237] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.600731 22442 leader_election.cc:288] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:54.601579 22444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.601758 22444 raft_consensus.cc:482] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.601858 22444 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.604349 22444 raft_consensus.cc:504] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.605401 22445 raft_consensus.cc:1226] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Rejecting Update request from peer 00c39ff1d7004a8f8a8847aad597f242 for earlier term 1. Current term is 2. Ops: []
I20240502 14:15:54.605687 22444 leader_election.cc:288] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:54.606031 24712 consensus_queue.cc:1046] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:54.606148 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.606341 22576 raft_consensus.cc:3032] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.606410 22576 raft_consensus.cc:729] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:54.606653 22576 consensus_queue.cc:260] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.606961 22576 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.609949 22444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.610054 22576 raft_consensus.cc:2454] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.610157 22444 raft_consensus.cc:482] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.610265 22444 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
W20240502 14:15:54.610344 24712 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20240502 14:15:54.611182 22379 leader_election.cc:302] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:54.612013 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.612277 22312 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.612731 22444 raft_consensus.cc:504] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.613346 22444 leader_election.cc:288] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:54.614145 22445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.614339 22445 raft_consensus.cc:482] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.614427 22445 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.616739 22445 raft_consensus.cc:504] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.617723 22445 leader_election.cc:288] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:54.618574 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:54.618758 22706 raft_consensus.cc:3032] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.618836 22706 raft_consensus.cc:729] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.618996 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.619079 22706 consensus_queue.cc:260] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.619207 22310 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.619438 22706 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.618574 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:54.619771 22708 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.620110 22312 raft_consensus.cc:2454] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.620222 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.620455 22311 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.620785 24718 raft_consensus.cc:686] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.621094 24718 consensus_queue.cc:237] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.622359 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.622438 22310 raft_consensus.cc:2454] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.622550 22575 raft_consensus.cc:3032] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.622611 22575 raft_consensus.cc:729] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:54.622941 22311 raft_consensus.cc:2454] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.622973 22377 leader_election.cc:302] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:54.622918 22575 consensus_queue.cc:260] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.623229 22575 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.626044 24728 raft_consensus.cc:2781] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.626494 24078 raft_consensus.cc:2781] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.626622 24078 raft_consensus.cc:686] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.626924 24078 consensus_queue.cc:237] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.628237 22708 raft_consensus.cc:2454] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.629766 22706 raft_consensus.cc:2454] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.630335 22442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.630586 22442 raft_consensus.cc:482] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.630770 22442 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.631429 24728 raft_consensus.cc:686] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.631973 24728 consensus_queue.cc:237] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.634619 22442 raft_consensus.cc:504] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.635352 22377 leader_election.cc:302] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.635479 22575 raft_consensus.cc:2454] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.635744 24730 raft_consensus.cc:2781] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.635938 24730 raft_consensus.cc:686] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.636042 22707 raft_consensus.cc:2454] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.636211 24730 consensus_queue.cc:237] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.639704 24444 heartbeater.cc:503] Master 127.6.196.254:33767 was elected leader, sending a full tablet report...
I20240502 14:15:54.640609 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.641059 22442 leader_election.cc:288] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:54.641167 22576 raft_consensus.cc:3032] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.641274 22576 raft_consensus.cc:729] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:54.641302 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.641498 22311 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.641487 24727 raft_consensus.cc:686] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.641538 22576 consensus_queue.cc:260] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.641918 22576 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.641768 24727 consensus_queue.cc:237] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.644050 22311 raft_consensus.cc:2454] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.645607 22377 leader_election.cc:302] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:54.645866 22576 raft_consensus.cc:2454] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.645942 24734 raft_consensus.cc:2781] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.646466 22442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.646639 22442 raft_consensus.cc:465] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 14:15:54.647440 22445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.647653 22445 raft_consensus.cc:482] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.647778 22445 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.652561 22166 catalog_manager.cc:5484] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:54.653096 22166 catalog_manager.cc:5484] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.653558 22166 catalog_manager.cc:5484] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:54.653954 22166 catalog_manager.cc:5484] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.656562 22445 raft_consensus.cc:504] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.658211 22445 leader_election.cc:288] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:54.658129 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:54.658376 22707 raft_consensus.cc:3032] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.658457 22707 raft_consensus.cc:729] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.658741 22707 consensus_queue.cc:260] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.658828 22575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.659091 22707 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.659082 22445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.659117 22575 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.659267 22445 raft_consensus.cc:482] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.659397 22445 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.661638 22575 raft_consensus.cc:2454] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.661908 22445 raft_consensus.cc:504] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.662657 22445 leader_election.cc:288] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:54.662895 22707 raft_consensus.cc:2454] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.663607 22442 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.663635 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.663810 22442 raft_consensus.cc:465] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 14:15:54.663859 22311 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.665503 22444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.665674 22444 raft_consensus.cc:482] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.665756 22444 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.667292 22311 raft_consensus.cc:2454] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.668121 22444 raft_consensus.cc:504] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.668681 22444 leader_election.cc:288] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:54.669697 22444 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd6972bee4d94830ac2d6a5660adc944"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.669905 22444 raft_consensus.cc:482] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.670069 22444 raft_consensus.cc:3037] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.672698 22444 raft_consensus.cc:504] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.673328 22444 leader_election.cc:288] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:54.673532 22377 leader_election.cc:302] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.674005 22377 leader_election.cc:302] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:54.675336 24078 raft_consensus.cc:2781] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.675206 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6972bee4d94830ac2d6a5660adc944" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.675617 24078 raft_consensus.cc:686] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.675944 24078 consensus_queue.cc:237] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.676985 24730 raft_consensus.cc:2781] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.677155 24730 raft_consensus.cc:686] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.677436 24730 consensus_queue.cc:237] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.675549 22311 raft_consensus.cc:3037] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.679332 22445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:54.679517 22445 raft_consensus.cc:482] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.679641 22445 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.679306 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:54.682474 22445 raft_consensus.cc:504] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.682405 22707 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.683389 24734 raft_consensus.cc:686] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.683640 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.683722 24734 consensus_queue.cc:237] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.683858 22310 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.685173 22445 leader_election.cc:288] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:54.685897 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:54.686118 22706 raft_consensus.cc:3032] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.686218 22706 raft_consensus.cc:729] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.686614 22706 consensus_queue.cc:260] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.688756 22706 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.690995 22310 raft_consensus.cc:2454] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.691413 22377 leader_election.cc:302] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:54.691706 24742 raft_consensus.cc:2781] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.691982 24742 raft_consensus.cc:686] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.693390 22706 raft_consensus.cc:2454] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.662982 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.669127 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.693909 22574 raft_consensus.cc:3032] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.694016 22574 raft_consensus.cc:729] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:54.692366 24742 consensus_queue.cc:237] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.694273 22574 consensus_queue.cc:260] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.694604 22574 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.696575 22574 raft_consensus.cc:2454] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.673992 22573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6972bee4d94830ac2d6a5660adc944" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.696815 22573 raft_consensus.cc:3032] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.696899 22573 raft_consensus.cc:729] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:54.697096 22573 consensus_queue.cc:260] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.697404 22573 raft_consensus.cc:3037] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.697436 22379 leader_election.cc:302] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:54.697662 24742 raft_consensus.cc:2781] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.697908 24742 raft_consensus.cc:686] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.698119 22576 raft_consensus.cc:3032] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.698235 22576 raft_consensus.cc:729] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:54.698168 24742 consensus_queue.cc:237] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.698454 22576 consensus_queue.cc:260] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.698763 22576 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.699864 22573 raft_consensus.cc:2454] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.700610 22379 leader_election.cc:302] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:54.701119 24734 raft_consensus.cc:2781] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.701318 24734 raft_consensus.cc:686] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.701525 24734 consensus_queue.cc:237] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.701884 22576 raft_consensus.cc:2454] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.703562 22311 raft_consensus.cc:2454] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.705842 22166 catalog_manager.cc:5484] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.706976 22707 raft_consensus.cc:2454] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 2.
I20240502 14:15:54.707489 22166 catalog_manager.cc:5484] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.708420 22166 catalog_manager.cc:5484] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.709488 22166 catalog_manager.cc:5484] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.709942 22166 catalog_manager.cc:5484] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:54.711691 22166 catalog_manager.cc:5484] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.712184 22166 catalog_manager.cc:5484] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.712652 22166 catalog_manager.cc:5484] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.722244 22166 catalog_manager.cc:5484] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.722966 22166 catalog_manager.cc:5484] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:54.723444 22166 catalog_manager.cc:5484] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.724006 22166 catalog_manager.cc:5484] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:54.724510 22166 catalog_manager.cc:5484] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 1 to 2, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 2 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:54.769325 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.769551 22707 raft_consensus.cc:482] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.769675 22707 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.771633 22707 raft_consensus.cc:504] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.772363 22707 leader_election.cc:288] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:54.772771 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.773000 22445 raft_consensus.cc:3032] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:54.773072 22445 raft_consensus.cc:729] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.773046 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.773293 22576 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.773393 22445 consensus_queue.cc:260] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.773788 22445 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.775183 22576 raft_consensus.cc:2454] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 3.
I20240502 14:15:54.775465 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.775629 22707 raft_consensus.cc:482] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.775691 22707 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.776715 22445 raft_consensus.cc:2396] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:54.777290 22707 raft_consensus.cc:504] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.777446 22642 leader_election.cc:302] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters: 017239f33db84a7b98c8c0244d55ecc5
I20240502 14:15:54.777778 24467 raft_consensus.cc:2781] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:54.777952 24467 raft_consensus.cc:686] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.778352 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.778267 24467 consensus_queue.cc:237] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.778508 22576 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.778544 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.778786 22445 raft_consensus.cc:3032] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:54.778851 22445 raft_consensus.cc:729] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.779127 22445 consensus_queue.cc:260] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.779372 22445 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.781517 22576 raft_consensus.cc:2454] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 3.
I20240502 14:15:54.781899 22445 raft_consensus.cc:2396] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:54.781879 22642 leader_election.cc:302] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.782049 22707 leader_election.cc:288] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:54.782193 24736 raft_consensus.cc:2781] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:54.783109 22706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.783341 22706 raft_consensus.cc:482] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.783488 22706 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.787019 22706 raft_consensus.cc:504] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.789384 24736 raft_consensus.cc:686] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.789650 24736 consensus_queue.cc:237] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.794471 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.794703 22311 raft_consensus.cc:3032] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.794811 22311 raft_consensus.cc:729] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:54.794811 22706 leader_election.cc:288] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:54.795070 22311 consensus_queue.cc:260] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.795228 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.795445 22576 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.795498 22311 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.795480 22166 catalog_manager.cc:5484] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:54.797811 22576 raft_consensus.cc:2454] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:54.798198 22642 leader_election.cc:302] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.798835 22311 raft_consensus.cc:2454] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:54.799079 24736 raft_consensus.cc:2781] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.799371 22706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.799548 22706 raft_consensus.cc:482] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.799649 22706 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.800387 24736 raft_consensus.cc:686] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.800644 24736 consensus_queue.cc:237] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.801883 22166 catalog_manager.cc:5484] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.804467 22706 raft_consensus.cc:504] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.805706 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.806298 22445 raft_consensus.cc:3032] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:54.806479 22445 raft_consensus.cc:729] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.806797 22445 consensus_queue.cc:260] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.807068 22445 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.809752 22445 raft_consensus.cc:2396] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:54.813835 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.813984 22706 leader_election.cc:288] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:54.814016 22576 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.815245 22706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f79b02ca45b4dea90513712bc57e137"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.815481 22706 raft_consensus.cc:482] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.815627 22706 raft_consensus.cc:3037] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.815611 22166 catalog_manager.cc:5484] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 1 to 2, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 2 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.816586 22576 raft_consensus.cc:2454] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 3.
I20240502 14:15:54.817004 22642 leader_election.cc:302] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters: 017239f33db84a7b98c8c0244d55ecc5
I20240502 14:15:54.817458 24736 raft_consensus.cc:2781] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:54.817704 24736 raft_consensus.cc:686] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.817962 24736 consensus_queue.cc:237] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.818557 22706 raft_consensus.cc:504] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.820197 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79b02ca45b4dea90513712bc57e137" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.820446 22576 raft_consensus.cc:3032] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.820523 22576 raft_consensus.cc:729] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:54.820802 22576 consensus_queue.cc:260] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.821486 22576 raft_consensus.cc:3037] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.824723 22576 raft_consensus.cc:2454] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:54.825213 22642 leader_election.cc:302] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.825515 24753 raft_consensus.cc:2781] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.825657 24753 raft_consensus.cc:686] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.825937 24753 consensus_queue.cc:237] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.826330 22706 leader_election.cc:288] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:54.827173 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79b02ca45b4dea90513712bc57e137" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.827401 22311 raft_consensus.cc:3037] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.827399 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.827551 22707 raft_consensus.cc:482] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.827658 22707 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.829947 22707 raft_consensus.cc:504] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.830094 22311 raft_consensus.cc:2454] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:54.830504 22707 leader_election.cc:288] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:54.832490 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44f40d47e54a429a9f55ae410e70ddf3"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.832684 22707 raft_consensus.cc:465] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 14:15:54.833753 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.833948 22707 raft_consensus.cc:482] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.834071 22707 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.836082 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.836365 22445 raft_consensus.cc:3032] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:54.836501 22445 raft_consensus.cc:729] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.836867 22445 consensus_queue.cc:260] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.837373 22445 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.841874 22445 raft_consensus.cc:2396] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:54.841060 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.843503 22576 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.843812 22166 catalog_manager.cc:5484] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.845366 22707 raft_consensus.cc:504] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.846415 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.846635 22311 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.849169 22311 raft_consensus.cc:2454] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 3.
I20240502 14:15:54.849825 22707 leader_election.cc:288] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:54.849874 22640 leader_election.cc:302] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.851120 22576 raft_consensus.cc:2454] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 3.
I20240502 14:15:54.851780 22642 leader_election.cc:302] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters: 017239f33db84a7b98c8c0244d55ecc5
I20240502 14:15:54.852836 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.853096 22445 raft_consensus.cc:3032] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:54.853202 22445 raft_consensus.cc:729] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.853446 24736 raft_consensus.cc:2781] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:54.853646 24467 raft_consensus.cc:2781] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:54.857039 22166 catalog_manager.cc:5484] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 1 to 2, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 2 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.861481 24467 raft_consensus.cc:686] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.861811 24467 consensus_queue.cc:237] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.862337 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.862509 22707 raft_consensus.cc:482] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.862653 22707 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.866654 24736 raft_consensus.cc:686] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.867025 24736 consensus_queue.cc:237] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.867816 22445 consensus_queue.cc:260] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.868312 22445 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.873979 22707 raft_consensus.cc:504] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.876319 22445 raft_consensus.cc:2396] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:54.877314 22707 leader_election.cc:288] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:54.877907 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.878132 22311 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.879402 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.879666 22707 raft_consensus.cc:482] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.879848 22707 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.880335 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.880539 22445 raft_consensus.cc:3032] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:54.880666 22311 raft_consensus.cc:2454] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 3.
I20240502 14:15:54.880735 22445 raft_consensus.cc:729] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.881088 22445 consensus_queue.cc:260] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.881433 22445 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.882405 22166 catalog_manager.cc:5484] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:54.883018 22166 catalog_manager.cc:5484] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.883342 22707 raft_consensus.cc:504] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.884061 22445 raft_consensus.cc:2396] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:54.884553 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.884694 22707 leader_election.cc:288] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:54.884766 22576 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.885716 22706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.885759 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.885917 22706 raft_consensus.cc:482] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.885949 22445 raft_consensus.cc:3032] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:54.886021 22706 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.886029 22445 raft_consensus.cc:729] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.886281 22445 consensus_queue.cc:260] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.886557 22445 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.887399 22576 raft_consensus.cc:2454] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 3.
I20240502 14:15:54.887976 22642 leader_election.cc:302] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.888413 22706 raft_consensus.cc:504] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.889411 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.889559 24467 raft_consensus.cc:2781] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:54.889607 22444 raft_consensus.cc:3032] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:54.889681 22444 raft_consensus.cc:729] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.889776 24467 raft_consensus.cc:686] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.889935 22444 consensus_queue.cc:260] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.890020 24467 consensus_queue.cc:237] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.890198 22444 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.890259 22445 raft_consensus.cc:2396] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:54.890762 22640 leader_election.cc:302] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters: 017239f33db84a7b98c8c0244d55ecc5
I20240502 14:15:54.891039 24736 raft_consensus.cc:2781] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:54.891183 24736 raft_consensus.cc:686] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.891413 24736 consensus_queue.cc:237] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.891870 22706 leader_election.cc:288] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:54.892619 22444 raft_consensus.cc:2396] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:54.893929 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.894132 22310 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.894346 22706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.894526 22706 raft_consensus.cc:482] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.894661 22706 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.897557 22706 raft_consensus.cc:504] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.897925 22310 raft_consensus.cc:2454] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 3.
I20240502 14:15:54.898510 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.898747 22444 raft_consensus.cc:3032] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:54.899017 22706 leader_election.cc:288] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:54.899150 22444 raft_consensus.cc:729] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.900167 22444 consensus_queue.cc:260] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.900614 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a53d97bb173340c586fe577b79ca5a55"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.900815 22707 raft_consensus.cc:482] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.901043 22444 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.901255 22707 raft_consensus.cc:3037] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.903482 22640 leader_election.cc:302] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters: 017239f33db84a7b98c8c0244d55ecc5
I20240502 14:15:54.904343 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.904626 22576 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.906024 22707 raft_consensus.cc:504] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.906684 22444 raft_consensus.cc:2396] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:54.907377 22576 raft_consensus.cc:2454] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 3.
I20240502 14:15:54.907799 22642 leader_election.cc:302] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters: 017239f33db84a7b98c8c0244d55ecc5
I20240502 14:15:54.908412 22707 leader_election.cc:288] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:54.908473 22573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53d97bb173340c586fe577b79ca5a55" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.908638 22573 raft_consensus.cc:3032] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.908735 22573 raft_consensus.cc:729] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:54.908986 22573 consensus_queue.cc:260] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.909386 22573 raft_consensus.cc:3037] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.909538 22706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.909783 22706 raft_consensus.cc:482] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.909942 22706 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.912242 22706 raft_consensus.cc:504] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.913234 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.913429 22706 leader_election.cc:288] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:54.913939 22573 raft_consensus.cc:2454] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:54.914345 22444 raft_consensus.cc:3032] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:54.914330 22708 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4c8f2b775fa54c44a5d855d586576696"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.914542 22708 raft_consensus.cc:465] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 14:15:54.914803 24753 raft_consensus.cc:2781] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:54.914994 24753 raft_consensus.cc:686] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.915066 24758 raft_consensus.cc:2781] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:54.914567 22444 raft_consensus.cc:729] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.915334 24753 consensus_queue.cc:237] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.915838 22444 consensus_queue.cc:260] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.916211 22444 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.916525 22708 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.916724 22708 raft_consensus.cc:482] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.916829 22708 raft_consensus.cc:3037] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.919205 22444 raft_consensus.cc:2396] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:54.919507 22708 raft_consensus.cc:504] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.920125 22708 leader_election.cc:288] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:54.920678 24758 raft_consensus.cc:686] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.920961 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53d97bb173340c586fe577b79ca5a55" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.921087 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.921042 24758 consensus_queue.cc:237] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:54.921249 22310 raft_consensus.cc:3037] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.921386 22311 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.922282 22642 leader_election.cc:302] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.922632 24753 raft_consensus.cc:2781] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.922731 24753 raft_consensus.cc:686] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.922932 24753 consensus_queue.cc:237] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:54.923878 22311 raft_consensus.cc:2454] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 3.
I20240502 14:15:54.923970 22310 raft_consensus.cc:2454] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:54.924612 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.924791 22574 raft_consensus.cc:3037] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.925753 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.925923 22444 raft_consensus.cc:3032] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.925987 22444 raft_consensus.cc:729] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.926208 22444 consensus_queue.cc:260] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.926493 22444 raft_consensus.cc:3037] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.927306 22574 raft_consensus.cc:2454] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:54.927697 22640 leader_election.cc:302] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters: 017239f33db84a7b98c8c0244d55ecc5
I20240502 14:15:54.928696 24758 raft_consensus.cc:2781] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:54.928743 22642 leader_election.cc:302] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.928859 24758 raft_consensus.cc:686] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.929092 24758 consensus_queue.cc:237] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.929407 22444 raft_consensus.cc:2454] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:54.930140 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.930310 22707 raft_consensus.cc:482] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.930398 22707 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.930150 22166 catalog_manager.cc:5484] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.930732 22166 catalog_manager.cc:5484] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.932108 22708 raft_consensus.cc:1261] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:54.932838 24742 consensus_queue.cc:1035] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:54.932931 22707 raft_consensus.cc:504] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.933629 24736 raft_consensus.cc:2781] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.933766 24736 raft_consensus.cc:686] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.933969 24736 consensus_queue.cc:237] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.935771 22707 leader_election.cc:288] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:54.936123 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:54.936298 22574 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.936864 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.937011 22707 raft_consensus.cc:482] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.937124 22707 raft_consensus.cc:3037] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.939121 22574 raft_consensus.cc:2454] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 3.
I20240502 14:15:54.939546 22642 leader_election.cc:302] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.939795 24753 raft_consensus.cc:2781] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:54.939992 24753 raft_consensus.cc:686] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.940114 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.940279 22444 raft_consensus.cc:3032] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:54.940192 24753 consensus_queue.cc:237] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.940342 22444 raft_consensus.cc:729] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.941393 22444 consensus_queue.cc:260] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.942057 22444 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.943310 22707 raft_consensus.cc:504] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.943845 22707 leader_election.cc:288] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:54.944190 22310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.944389 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.944439 22310 raft_consensus.cc:3037] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.944579 22445 raft_consensus.cc:3032] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:54.944661 22445 raft_consensus.cc:729] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.944967 22445 consensus_queue.cc:260] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.945463 22445 raft_consensus.cc:3037] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:54.947067 22310 raft_consensus.cc:2454] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:54.948654 22445 raft_consensus.cc:2454] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:54.949196 22640 leader_election.cc:302] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:54.949201 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:54.950320 22574 raft_consensus.cc:1261] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:54.950892 24763 raft_consensus.cc:2781] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:54.950868 24742 consensus_queue.cc:1035] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:54.951001 24763 raft_consensus.cc:686] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.951234 24763 consensus_queue.cc:237] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:54.954186 22444 raft_consensus.cc:2396] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:54.966768 22310 raft_consensus.cc:1261] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:54.967533 24078 consensus_queue.cc:1035] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:54.979559 22707 raft_consensus.cc:482] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:54.981266 22707 raft_consensus.cc:3037] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.983641 22311 raft_consensus.cc:1261] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:54.984236 24734 consensus_queue.cc:1035] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:54.984884 22707 raft_consensus.cc:504] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.985927 22707 leader_election.cc:288] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:54.987354 22311 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:54.989122 22166 catalog_manager.cc:5484] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 1 to 2, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 2 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.989775 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:54.989987 22444 raft_consensus.cc:3032] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:54.990063 22444 raft_consensus.cc:729] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:54.989857 22166 catalog_manager.cc:5484] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:54.990289 22444 consensus_queue.cc:260] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:54.990573 22444 raft_consensus.cc:3037] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:54.990909 22166 catalog_manager.cc:5484] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 1 to 2, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 2 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.991448 22166 catalog_manager.cc:5484] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.991951 22166 catalog_manager.cc:5484] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:54.992441 22166 catalog_manager.cc:5484] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:54.992991 22444 raft_consensus.cc:2454] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 3.
I20240502 14:15:54.993513 22643 leader_election.cc:302] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters: 2b54c913a94f4c399ec08cf222a941e6
I20240502 14:15:54.992961 22166 catalog_manager.cc:5484] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 1 to 2, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 2 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:54.994643 24763 raft_consensus.cc:2781] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:54.997653 24763 raft_consensus.cc:686] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:54.998291 24763 consensus_queue.cc:237] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.001189 22706 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:55.003649 22706 raft_consensus.cc:482] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.003963 22706 raft_consensus.cc:3037] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.008350 22169 catalog_manager.cc:5484] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:55.009290 22706 raft_consensus.cc:504] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.012204 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.012416 22444 raft_consensus.cc:3032] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:55.012493 22444 raft_consensus.cc:729] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:55.012729 22444 consensus_queue.cc:260] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.013095 22444 raft_consensus.cc:3037] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.014551 22706 leader_election.cc:288] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:55.015049 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:55.015292 22312 raft_consensus.cc:3037] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.015535 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9da0fcab2294a3fb62c132a6b60214c"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:55.015683 22707 raft_consensus.cc:482] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.015789 22707 raft_consensus.cc:3037] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.018047 22312 raft_consensus.cc:2454] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:55.018214 22707 raft_consensus.cc:504] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.018442 22640 leader_election.cc:302] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:55.019122 24763 raft_consensus.cc:2781] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:55.020627 22444 raft_consensus.cc:2454] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:55.020825 24763 raft_consensus.cc:686] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.021128 24763 consensus_queue.cc:237] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.026151 22707 leader_election.cc:288] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:55.026808 22309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9da0fcab2294a3fb62c132a6b60214c" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:55.026976 22309 raft_consensus.cc:3037] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.030027 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7951641f78914cc583da7f0c4b99ed9d"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:55.030079 22312 raft_consensus.cc:1261] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.030241 22707 raft_consensus.cc:465] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Not starting forced leader election -- already a leader
I20240502 14:15:55.031483 22707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4"
dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
from {username='slave'} at 127.0.0.1:43732
I20240502 14:15:55.031673 22707 raft_consensus.cc:482] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.031636 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9da0fcab2294a3fb62c132a6b60214c" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.031781 22707 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.031867 22574 raft_consensus.cc:3032] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:55.031960 22574 raft_consensus.cc:729] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:55.031931 22309 raft_consensus.cc:2454] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:55.032274 22574 consensus_queue.cc:260] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.032580 22640 leader_election.cc:302] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:55.032713 22574 raft_consensus.cc:3037] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.033037 24753 raft_consensus.cc:2781] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:55.033185 24753 raft_consensus.cc:686] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.033463 24753 consensus_queue.cc:237] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.030881 24734 consensus_queue.cc:1035] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.035245 22707 raft_consensus.cc:504] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.036379 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:55.036644 22574 raft_consensus.cc:2454] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 2.
I20240502 14:15:55.036721 22308 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.037967 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.038198 22444 raft_consensus.cc:3032] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.038283 22444 raft_consensus.cc:729] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:55.038594 22444 consensus_queue.cc:260] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.038906 22444 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.039330 24578 heartbeater.cc:503] Master 127.6.196.254:33767 was elected leader, sending a full tablet report...
I20240502 14:15:55.040565 22707 leader_election.cc:288] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:55.044812 22444 raft_consensus.cc:2396] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.051260 22308 raft_consensus.cc:2454] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 3.
I20240502 14:15:55.052522 22309 raft_consensus.cc:1261] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.053252 24734 consensus_queue.cc:1035] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.056840 22166 catalog_manager.cc:5484] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 1 to 2, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 2 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:55.063005 22640 leader_election.cc:302] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters: 017239f33db84a7b98c8c0244d55ecc5
I20240502 14:15:55.064909 22166 catalog_manager.cc:5484] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 1 to 2, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 2 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.069082 24763 raft_consensus.cc:2781] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.069319 24763 raft_consensus.cc:686] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.070286 24763 consensus_queue.cc:237] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.077906 22308 raft_consensus.cc:1261] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.078907 22309 raft_consensus.cc:1261] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.079116 22166 catalog_manager.cc:5484] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 3 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.079437 24734 consensus_queue.cc:1035] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.082039 24734 consensus_queue.cc:1035] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.085490 22574 raft_consensus.cc:1261] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.086170 24078 consensus_queue.cc:1035] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.110971 22573 raft_consensus.cc:1261] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.111528 24730 consensus_queue.cc:1035] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.112713 22576 raft_consensus.cc:1261] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.113971 24742 consensus_queue.cc:1035] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.115458 22575 raft_consensus.cc:1261] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.118472 24734 consensus_queue.cc:1035] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.132854 22309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.133147 22309 raft_consensus.cc:482] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.133307 22309 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.136265 22309 raft_consensus.cc:504] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.137167 22309 leader_election.cc:288] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:55.137641 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.137835 22574 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.138026 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.138245 22707 raft_consensus.cc:3032] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.138362 22707 raft_consensus.cc:729] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.138644 22707 consensus_queue.cc:260] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.139024 22707 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.140621 22574 raft_consensus.cc:2454] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.140991 22247 leader_election.cc:302] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:55.141245 22707 raft_consensus.cc:2396] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.141332 24793 raft_consensus.cc:2781] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.141487 24793 raft_consensus.cc:686] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.141920 22309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.141867 24793 consensus_queue.cc:237] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.142092 22309 raft_consensus.cc:482] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.142228 22309 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.144609 22309 raft_consensus.cc:504] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.145762 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.145505 22166 catalog_manager.cc:5484] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 2 to 3, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:55.146078 22574 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.146234 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.146499 22444 raft_consensus.cc:3032] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.146608 22444 raft_consensus.cc:729] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:55.147003 22444 consensus_queue.cc:260] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.147357 22444 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.149885 22574 raft_consensus.cc:2454] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.150328 22247 leader_election.cc:302] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:55.150585 24793 raft_consensus.cc:2781] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.150624 22444 raft_consensus.cc:2454] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.150738 24793 raft_consensus.cc:686] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.151031 22309 leader_election.cc:288] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:55.150969 24793 consensus_queue.cc:237] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.152158 22309 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.152315 22309 raft_consensus.cc:482] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.152403 22309 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.154048 22166 catalog_manager.cc:5484] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.154780 22309 raft_consensus.cc:504] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.155735 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.155926 22444 raft_consensus.cc:3032] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.156013 22444 raft_consensus.cc:729] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:55.156129 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.156344 22574 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.156267 22444 consensus_queue.cc:260] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.156616 22444 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.158703 22444 raft_consensus.cc:2454] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.159039 22248 leader_election.cc:302] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:55.159240 22309 leader_election.cc:288] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:55.159299 24316 raft_consensus.cc:2781] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.159431 24316 raft_consensus.cc:686] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.159693 24316 consensus_queue.cc:237] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.160105 22574 raft_consensus.cc:2454] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.160264 22308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f79b02ca45b4dea90513712bc57e137"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.160452 22308 raft_consensus.cc:482] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.160566 22308 raft_consensus.cc:3037] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.162355 22308 raft_consensus.cc:504] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.162834 22308 leader_election.cc:288] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.163693 22308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44f40d47e54a429a9f55ae410e70ddf3"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.163842 22308 raft_consensus.cc:482] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.163930 22308 raft_consensus.cc:3037] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.164530 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79b02ca45b4dea90513712bc57e137" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.164746 22707 raft_consensus.cc:3032] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.164857 22707 raft_consensus.cc:729] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.165164 22707 consensus_queue.cc:260] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.165518 22707 raft_consensus.cc:3037] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.166217 22308 raft_consensus.cc:504] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.166453 22166 catalog_manager.cc:5484] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:55.167208 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40d47e54a429a9f55ae410e70ddf3" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.167407 22576 raft_consensus.cc:3037] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.167928 22707 raft_consensus.cc:2396] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.169453 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79b02ca45b4dea90513712bc57e137" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.169580 22573 raft_consensus.cc:1261] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.169653 22574 raft_consensus.cc:3037] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.169842 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40d47e54a429a9f55ae410e70ddf3" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.170085 22706 raft_consensus.cc:3032] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:55.170198 22706 raft_consensus.cc:729] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.170459 22706 consensus_queue.cc:260] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.170795 22706 raft_consensus.cc:3037] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.170791 24742 consensus_queue.cc:1035] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.171934 22574 raft_consensus.cc:2454] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.172369 22247 leader_election.cc:302] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters: c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:55.172761 24316 raft_consensus.cc:2781] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.172915 24316 raft_consensus.cc:686] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.173278 24316 consensus_queue.cc:237] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.169664 22576 raft_consensus.cc:2454] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 2.
I20240502 14:15:55.174010 22247 leader_election.cc:302] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:55.174350 24793 raft_consensus.cc:2781] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:55.174489 24793 raft_consensus.cc:686] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.174547 22706 raft_consensus.cc:2454] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 2.
I20240502 14:15:55.174744 24793 consensus_queue.cc:237] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.177104 22308 leader_election.cc:288] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 2 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.182487 22307 raft_consensus.cc:1261] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.182634 22309 raft_consensus.cc:1261] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.184700 24734 consensus_queue.cc:1035] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.185286 24742 consensus_queue.cc:1035] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 14:15:55.190532 22308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.190737 22308 raft_consensus.cc:482] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.190879 22308 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.194380 22308 raft_consensus.cc:504] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.195689 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.195956 22444 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.199283 22444 raft_consensus.cc:2396] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.205744 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.206012 22706 raft_consensus.cc:3032] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.206126 22706 raft_consensus.cc:729] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.206470 22706 consensus_queue.cc:260] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.206810 22706 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.210220 22706 raft_consensus.cc:2396] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.210755 22245 leader_election.cc:302] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6; no voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:55.215600 24807 raft_consensus.cc:2726] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 14:15:55.217885 22308 leader_election.cc:288] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 4 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.219354 22312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.219635 22312 raft_consensus.cc:482] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.219763 22312 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.223268 22312 raft_consensus.cc:504] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.224177 22312 leader_election.cc:288] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 4 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.224741 22166 catalog_manager.cc:5484] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 2 to 3, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:55.225852 22312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.226096 22312 raft_consensus.cc:482] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.226294 22312 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.231204 22312 raft_consensus.cc:504] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.231694 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.231925 22444 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.236091 22444 raft_consensus.cc:2396] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.237700 22312 leader_election.cc:288] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:55.238009 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.238147 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.238345 22576 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.238970 22706 raft_consensus.cc:3032] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.239248 22706 raft_consensus.cc:729] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.239725 22706 consensus_queue.cc:260] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.240217 22706 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.242023 22576 raft_consensus.cc:2454] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.242372 22247 leader_election.cc:302] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:55.243525 24809 raft_consensus.cc:2781] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.243697 24809 raft_consensus.cc:686] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.244027 24809 consensus_queue.cc:237] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.244616 22312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.244868 22312 raft_consensus.cc:482] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.245041 22312 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.251631 22573 raft_consensus.cc:1261] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Refusing update from remote peer 017239f33db84a7b98c8c0244d55ecc5: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.237099 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.251840 22445 raft_consensus.cc:3032] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.251914 22445 raft_consensus.cc:729] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:55.252128 22445 consensus_queue.cc:260] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.252444 22445 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.254686 22445 raft_consensus.cc:2454] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.254652 22166 catalog_manager.cc:5484] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 1 to 2, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 2 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.255635 24718 consensus_queue.cc:1035] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.255812 22312 raft_consensus.cc:504] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.256361 22312 leader_election.cc:288] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 4 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.257045 22706 raft_consensus.cc:2396] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.258350 22245 leader_election.cc:302] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6; no voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:55.258630 24793 raft_consensus.cc:2726] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 14:15:55.262462 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.262641 22444 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.264472 22166 catalog_manager.cc:5484] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:55.264711 22308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.264945 22308 raft_consensus.cc:482] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.265064 22308 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.266968 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.267256 22706 raft_consensus.cc:3032] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.267344 22706 raft_consensus.cc:729] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.269737 22308 raft_consensus.cc:504] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.270758 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.270956 22576 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.270961 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.271178 22445 raft_consensus.cc:3032] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.271265 22445 raft_consensus.cc:729] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:55.271584 22445 consensus_queue.cc:260] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.272070 22445 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.273604 22576 raft_consensus.cc:2454] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.273929 22247 leader_election.cc:302] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:55.274161 24793 raft_consensus.cc:2781] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.274328 24793 raft_consensus.cc:686] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.274583 24793 consensus_queue.cc:237] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.275367 22445 raft_consensus.cc:2454] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.277391 22706 consensus_queue.cc:260] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.277755 22706 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.278456 22444 raft_consensus.cc:2396] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.277520 22166 catalog_manager.cc:5484] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.280174 22308 leader_election.cc:288] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:55.282212 22706 raft_consensus.cc:2396] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.282599 22245 leader_election.cc:302] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6; no voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:55.282966 24809 raft_consensus.cc:2726] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 14:15:55.283160 22308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.283434 22308 raft_consensus.cc:482] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.283536 22308 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.286614 22308 raft_consensus.cc:504] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.287204 22308 leader_election.cc:288] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:55.287588 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.287777 22444 raft_consensus.cc:3032] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.287870 22444 raft_consensus.cc:729] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:55.288142 22444 consensus_queue.cc:260] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.288450 22308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a53d97bb173340c586fe577b79ca5a55"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.288480 22444 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.288643 22308 raft_consensus.cc:482] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.288764 22308 raft_consensus.cc:3037] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.289721 22573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.289914 22573 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.290849 22444 raft_consensus.cc:2454] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.291505 22248 leader_election.cc:302] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:55.291968 24809 raft_consensus.cc:2781] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.292095 24809 raft_consensus.cc:686] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.292309 22573 raft_consensus.cc:2454] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.292351 24809 consensus_queue.cc:237] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.292552 22308 raft_consensus.cc:504] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.293135 22308 leader_election.cc:288] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:55.293697 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53d97bb173340c586fe577b79ca5a55" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.293913 22706 raft_consensus.cc:3032] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.294008 22706 raft_consensus.cc:729] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.294323 22706 consensus_queue.cc:260] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.294616 22706 raft_consensus.cc:3037] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.294631 22573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53d97bb173340c586fe577b79ca5a55" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.294826 22573 raft_consensus.cc:3037] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.295022 22169 catalog_manager.cc:5484] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:55.296427 22308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.296576 22308 raft_consensus.cc:482] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.296687 22308 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.298584 22706 raft_consensus.cc:2396] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.299072 22308 raft_consensus.cc:504] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.299600 22308 leader_election.cc:288] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 4 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.300088 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.300331 22444 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.300451 22312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4c8f2b775fa54c44a5d855d586576696"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.300629 22312 raft_consensus.cc:482] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.300729 22312 raft_consensus.cc:3037] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.301326 22573 raft_consensus.cc:2454] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.301762 22247 leader_election.cc:302] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters: c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:55.302297 24793 raft_consensus.cc:2781] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.302428 24793 raft_consensus.cc:686] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.302623 24793 consensus_queue.cc:237] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.303748 22444 raft_consensus.cc:2396] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.304006 22445 raft_consensus.cc:1261] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 14:15:55.304124 22445 pending_rounds.cc:76] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5: Aborting all ops after (but not including) 3
I20240502 14:15:55.304215 22445 pending_rounds.cc:97] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 14:15:55.304368 22445 raft_consensus.cc:2866] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:55.300333 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.304973 24763 consensus_queue.cc:1035] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240502 14:15:55.305822 22706 raft_consensus.cc:3032] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.305912 22706 raft_consensus.cc:729] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.306149 22706 consensus_queue.cc:260] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.306437 22706 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.306994 22169 catalog_manager.cc:5484] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 2 to 3, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:55.308876 22444 raft_consensus.cc:1261] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 14:15:55.308912 22706 raft_consensus.cc:2396] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.308992 22444 pending_rounds.cc:76] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5: Aborting all ops after (but not including) 3
I20240502 14:15:55.309039 22444 pending_rounds.cc:97] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 14:15:55.309103 22444 raft_consensus.cc:2866] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:55.309739 24763 consensus_queue.cc:1035] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.311632 22573 raft_consensus.cc:1261] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 14:15:55.312144 24763 consensus_queue.cc:1035] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.312582 22245 leader_election.cc:302] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6; no voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:55.313040 24793 raft_consensus.cc:2726] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 14:15:55.313334 22576 raft_consensus.cc:1261] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 14:15:55.314962 24467 consensus_queue.cc:1035] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.322996 22312 raft_consensus.cc:504] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.323652 22308 raft_consensus.cc:1226] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Rejecting Update request from peer c8a8e31da9734e118e5095919ae17f10 for earlier term 1. Current term is 2. Ops: []
I20240502 14:15:55.324959 24753 consensus_queue.cc:1046] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.325841 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8f2b775fa54c44a5d855d586576696" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.326108 22574 raft_consensus.cc:3037] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.326552 22312 leader_election.cc:288] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 2 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.330843 22573 raft_consensus.cc:1261] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 14:15:55.330873 22312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.330905 22574 raft_consensus.cc:2454] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 2.
I20240502 14:15:55.331064 22312 raft_consensus.cc:482] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.331179 22312 raft_consensus.cc:3037] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.331292 22247 leader_election.cc:302] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:55.331595 24793 raft_consensus.cc:2781] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:55.331830 24793 raft_consensus.cc:686] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.331990 24758 consensus_queue.cc:1035] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.332098 24793 consensus_queue.cc:237] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.332899 24736 raft_consensus.cc:3032] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:55.332973 24736 raft_consensus.cc:729] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.333194 24736 consensus_queue.cc:260] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.333556 24736 raft_consensus.cc:3037] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.334667 22312 raft_consensus.cc:504] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.335634 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.335903 22445 raft_consensus.cc:2379] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 in current term 3: Already voted for candidate c8a8e31da9734e118e5095919ae17f10 in this term.
I20240502 14:15:55.336489 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8f2b775fa54c44a5d855d586576696" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.336885 22312 leader_election.cc:288] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.337006 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.337370 22707 raft_consensus.cc:2379] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 in current term 3: Already voted for candidate c8a8e31da9734e118e5095919ae17f10 in this term.
I20240502 14:15:55.337778 22245 leader_election.cc:302] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6; no voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:55.338043 24809 raft_consensus.cc:2726] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20240502 14:15:55.339025 22706 raft_consensus.cc:2454] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 2.
I20240502 14:15:55.339771 22312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.339949 22312 raft_consensus.cc:482] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.340087 22312 raft_consensus.cc:3037] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.341542 22445 raft_consensus.cc:1261] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 14:15:55.341651 22445 pending_rounds.cc:76] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5: Aborting all ops after (but not including) 3
I20240502 14:15:55.341703 22445 pending_rounds.cc:97] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 14:15:55.341770 22445 raft_consensus.cc:2866] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:55.342381 24736 consensus_queue.cc:1035] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.342470 22312 raft_consensus.cc:504] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.343824 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.343976 22444 raft_consensus.cc:3037] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.346244 22444 raft_consensus.cc:2454] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.346872 22248 leader_election.cc:302] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:55.347180 24809 raft_consensus.cc:2781] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.347344 24809 raft_consensus.cc:686] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.347688 24809 consensus_queue.cc:237] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.348224 22312 leader_election.cc:288] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.348946 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.349121 22706 raft_consensus.cc:3032] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.349191 22706 raft_consensus.cc:729] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.349432 22706 consensus_queue.cc:260] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.349520 22444 raft_consensus.cc:1261] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 14:15:55.349598 22444 pending_rounds.cc:76] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5: Aborting all ops after (but not including) 3
I20240502 14:15:55.349659 22444 pending_rounds.cc:97] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 14:15:55.349651 22706 raft_consensus.cc:3037] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.349710 22444 raft_consensus.cc:2866] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:55.351065 22312 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9da0fcab2294a3fb62c132a6b60214c"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.351274 22312 raft_consensus.cc:482] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.351704 22312 raft_consensus.cc:3037] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.351651 24753 consensus_queue.cc:1035] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.352234 22706 raft_consensus.cc:2396] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.353364 22442 raft_consensus.cc:1261] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 14:15:55.353471 22442 pending_rounds.cc:76] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5: Aborting all ops after (but not including) 3
I20240502 14:15:55.353514 22442 pending_rounds.cc:97] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 14:15:55.353574 22442 raft_consensus.cc:2866] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:55.354514 22312 raft_consensus.cc:504] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.355366 22573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9da0fcab2294a3fb62c132a6b60214c" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.355413 24753 consensus_queue.cc:1035] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.355700 22573 raft_consensus.cc:3037] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.357856 22312 leader_election.cc:288] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.358448 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9da0fcab2294a3fb62c132a6b60214c" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.358639 22706 raft_consensus.cc:3032] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.358726 22706 raft_consensus.cc:729] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.358695 22308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.358836 22308 raft_consensus.cc:482] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.358945 22308 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.358955 22706 consensus_queue.cc:260] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.359223 22706 raft_consensus.cc:3037] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.359447 22573 raft_consensus.cc:2454] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.361390 22247 leader_election.cc:302] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:55.361650 24810 raft_consensus.cc:2781] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.361723 22706 raft_consensus.cc:2396] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.361809 24810 raft_consensus.cc:686] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.362119 24810 consensus_queue.cc:237] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.363512 22308 raft_consensus.cc:504] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.364137 22308 leader_election.cc:288] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:55.364763 22574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.366714 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.366873 22444 raft_consensus.cc:3032] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.366950 22444 raft_consensus.cc:729] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:55.367161 22444 consensus_queue.cc:260] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.367494 22444 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.367300 22169 catalog_manager.cc:5484] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 1 to 2, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 2 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.367894 22576 raft_consensus.cc:1261] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 14:15:55.368383 22442 raft_consensus.cc:1261] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.368505 24736 consensus_queue.cc:1035] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.368992 24758 consensus_queue.cc:1035] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.369354 22308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.369515 22308 raft_consensus.cc:482] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.369624 22308 raft_consensus.cc:3037] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.375226 22308 raft_consensus.cc:504] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.376284 22574 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.376786 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.376988 22706 raft_consensus.cc:3032] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.377064 22706 raft_consensus.cc:729] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.377382 22706 consensus_queue.cc:260] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.377607 22706 raft_consensus.cc:3037] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.378388 22444 raft_consensus.cc:2454] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.378756 22248 leader_election.cc:302] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:55.379060 22574 raft_consensus.cc:2454] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.379453 24793 raft_consensus.cc:2781] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.379607 24793 raft_consensus.cc:686] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.379766 22573 raft_consensus.cc:1261] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 14:15:55.379887 24793 consensus_queue.cc:237] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.380354 24736 consensus_queue.cc:1035] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.381361 22706 raft_consensus.cc:2396] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.381944 22444 raft_consensus.cc:1261] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 14:15:55.382030 22444 pending_rounds.cc:76] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5: Aborting all ops after (but not including) 3
I20240502 14:15:55.382097 22444 pending_rounds.cc:97] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 14:15:55.382153 22444 raft_consensus.cc:2866] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:55.382560 24736 consensus_queue.cc:1035] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.383986 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.384207 22445 raft_consensus.cc:3037] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.386425 22576 raft_consensus.cc:1261] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 14:15:55.386718 22443 raft_consensus.cc:1261] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 14:15:55.386822 22443 pending_rounds.cc:76] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5: Aborting all ops after (but not including) 3
I20240502 14:15:55.386873 22443 pending_rounds.cc:97] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 14:15:55.386945 22443 raft_consensus.cc:2866] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:55.386917 24736 consensus_queue.cc:1035] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.387676 24736 consensus_queue.cc:1035] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.388695 22445 raft_consensus.cc:2454] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.389159 22248 leader_election.cc:302] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters: c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:55.389477 24793 raft_consensus.cc:2781] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.389622 24793 raft_consensus.cc:686] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.389820 22308 leader_election.cc:288] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:55.389865 24793 consensus_queue.cc:237] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.392174 22572 raft_consensus.cc:1261] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 14:15:55.392689 24753 consensus_queue.cc:1035] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.401576 22308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd6972bee4d94830ac2d6a5660adc944"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.401810 22308 raft_consensus.cc:482] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.401996 22308 raft_consensus.cc:3037] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.403477 22571 raft_consensus.cc:1261] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.404961 24736 consensus_queue.cc:1035] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.405045 22308 raft_consensus.cc:504] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.411629 22442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6972bee4d94830ac2d6a5660adc944" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.411859 22442 raft_consensus.cc:3032] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.411813 22572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6972bee4d94830ac2d6a5660adc944" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.412027 22442 raft_consensus.cc:729] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:55.412040 22572 raft_consensus.cc:3037] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.412313 22442 consensus_queue.cc:260] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.412585 22442 raft_consensus.cc:3037] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.414134 22442 raft_consensus.cc:2454] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.415340 22308 leader_election.cc:288] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:55.415342 22572 raft_consensus.cc:2454] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 3.
I20240502 14:15:55.415391 22248 leader_election.cc:302] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:55.415907 24793 raft_consensus.cc:2781] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.416061 24793 raft_consensus.cc:686] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.416903 24793 consensus_queue.cc:237] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.417363 22308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7951641f78914cc583da7f0c4b99ed9d"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.417552 22308 raft_consensus.cc:482] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.417660 22308 raft_consensus.cc:3037] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.419899 22308 raft_consensus.cc:504] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.420926 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7951641f78914cc583da7f0c4b99ed9d" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.421058 22442 raft_consensus.cc:1261] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.421142 22706 raft_consensus.cc:3032] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Stepping down as leader of term 1
I20240502 14:15:55.421216 22706 raft_consensus.cc:729] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.421640 22706 consensus_queue.cc:260] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.421610 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7951641f78914cc583da7f0c4b99ed9d" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:55.421976 22706 raft_consensus.cc:3037] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.421983 22576 raft_consensus.cc:3037] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 1 FOLLOWER]: Advancing to term 2
I20240502 14:15:55.423868 22706 raft_consensus.cc:2454] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 2.
I20240502 14:15:55.421639 24753 consensus_queue.cc:1035] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240502 14:15:55.424275 22245 leader_election.cc:302] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:55.424603 24809 raft_consensus.cc:2781] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Leader election won for term 2
I20240502 14:15:55.424826 22308 leader_election.cc:288] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 2 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.426369 22308 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4"
dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
from {username='slave'} at 127.0.0.1:54372
I20240502 14:15:55.426545 22308 raft_consensus.cc:482] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.426676 22308 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.429805 22312 raft_consensus.cc:1261] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Refusing update from remote peer c8a8e31da9734e118e5095919ae17f10: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.430394 24753 consensus_queue.cc:1035] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.434504 24809 raft_consensus.cc:686] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 2 LEADER]: Becoming Leader. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.434826 24809 consensus_queue.cc:237] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.432973 22308 raft_consensus.cc:504] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.438206 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.438475 22706 raft_consensus.cc:3032] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.438565 22706 raft_consensus.cc:729] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.438843 22706 consensus_queue.cc:260] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.439153 22706 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.442445 22165 catalog_manager.cc:5484] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 2 to 3, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.442756 22706 raft_consensus.cc:2396] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.443048 22165 catalog_manager.cc:5484] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 2 to 3, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:55.443357 22308 leader_election.cc:288] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 4 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:55.444226 22576 raft_consensus.cc:2454] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b54c913a94f4c399ec08cf222a941e6 in term 2.
I20240502 14:15:55.447165 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "2b54c913a94f4c399ec08cf222a941e6" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.447381 22445 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.450660 22445 raft_consensus.cc:2396] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 2b54c913a94f4c399ec08cf222a941e6 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.451967 22248 leader_election.cc:302] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2b54c913a94f4c399ec08cf222a941e6; no voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:55.453190 24810 raft_consensus.cc:2726] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 14:15:55.456383 22163 catalog_manager.cc:5484] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.456936 22163 catalog_manager.cc:5484] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 2 to 3, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.457437 22163 catalog_manager.cc:5484] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 1 to 2, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 2 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.458002 22163 catalog_manager.cc:5484] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 reported cstate change: term changed from 2 to 3, leader changed from 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194) to 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193). New cstate: current_term: 3 leader_uuid: "2b54c913a94f4c399ec08cf222a941e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:55.491930 24708 heartbeater.cc:503] Master 127.6.196.254:33767 was elected leader, sending a full tablet report...
I20240502 14:15:55.520787 22575 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:55.521054 22575 raft_consensus.cc:482] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.521167 22575 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.523905 22575 raft_consensus.cc:504] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.525430 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.525718 22445 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.526428 22575 leader_election.cc:288] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.527405 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.527637 22706 raft_consensus.cc:3032] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.527726 22706 raft_consensus.cc:729] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.528025 22706 consensus_queue.cc:260] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.528404 22706 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.529481 22445 raft_consensus.cc:2454] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.529870 22575 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:55.530031 22575 raft_consensus.cc:482] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.530138 22575 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.531385 22706 raft_consensus.cc:2454] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.531806 22509 leader_election.cc:302] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:55.532085 24712 raft_consensus.cc:2781] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:55.532670 24712 raft_consensus.cc:686] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:55.532860 22575 raft_consensus.cc:504] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.532997 24712 consensus_queue.cc:237] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.533727 22575 leader_election.cc:288] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:55.534025 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.534230 22706 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.535099 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:55.535321 22308 raft_consensus.cc:3032] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.535395 22308 raft_consensus.cc:729] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.535677 22308 consensus_queue.cc:260] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.536042 22308 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.536074 22166 catalog_manager.cc:5484] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.539184 22706 raft_consensus.cc:2396] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00c39ff1d7004a8f8a8847aad597f242 for term 4 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.540006 22308 raft_consensus.cc:2396] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00c39ff1d7004a8f8a8847aad597f242 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.540498 22509 leader_election.cc:302] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242; no voters: 2b54c913a94f4c399ec08cf222a941e6, c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:55.540938 24712 raft_consensus.cc:2726] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20240502 14:15:55.549355 22575 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:55.549784 22575 raft_consensus.cc:482] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.550086 22575 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.552827 22575 raft_consensus.cc:504] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:55.553535 22575 leader_election.cc:288] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.553984 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.554158 22445 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.554975 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.555202 22706 raft_consensus.cc:3032] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.555311 22706 raft_consensus.cc:729] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.555694 22706 consensus_queue.cc:260] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:55.556159 22706 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.557183 22445 raft_consensus.cc:2454] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.562407 22706 raft_consensus.cc:2454] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.557657 22512 leader_election.cc:302] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:55.558287 22576 raft_consensus.cc:1261] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 14:15:55.565361 24712 raft_consensus.cc:2781] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:55.565523 24712 raft_consensus.cc:686] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:55.565835 24712 consensus_queue.cc:237] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:55.566190 24809 consensus_queue.cc:1035] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.569972 22166 catalog_manager.cc:5484] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:55.574910 22555 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240502 14:15:55.581212 22571 raft_consensus.cc:1261] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.581852 24809 consensus_queue.cc:1035] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.581938 22575 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5f79b02ca45b4dea90513712bc57e137"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:55.591910 22575 raft_consensus.cc:482] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.593868 22706 raft_consensus.cc:1261] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 14:15:55.593984 22706 pending_rounds.cc:76] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10: Aborting all ops after (but not including) 3
I20240502 14:15:55.594072 22706 pending_rounds.cc:97] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 14:15:55.593891 22575 raft_consensus.cc:3037] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.594179 22706 raft_consensus.cc:2866] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:55.594763 24811 consensus_queue.cc:1035] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.599184 22422 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:55.607434 22576 raft_consensus.cc:1261] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.608024 24811 consensus_queue.cc:1035] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.612661 22445 raft_consensus.cc:1261] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.613587 24809 consensus_queue.cc:1035] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.630908 22575 raft_consensus.cc:504] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.651057 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79b02ca45b4dea90513712bc57e137" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.651243 22706 raft_consensus.cc:3037] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.651701 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f79b02ca45b4dea90513712bc57e137" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:55.651964 22308 raft_consensus.cc:3032] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.652067 22308 raft_consensus.cc:729] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.652367 22308 consensus_queue.cc:260] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.652769 22308 raft_consensus.cc:3037] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.653972 22706 raft_consensus.cc:2454] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.654371 22509 leader_election.cc:302] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:55.654657 24852 raft_consensus.cc:2781] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:55.654808 24852 raft_consensus.cc:686] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:55.655135 24852 consensus_queue.cc:237] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.656674 22308 raft_consensus.cc:2454] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.661319 22575 leader_election.cc:288] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:55.662611 22576 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "44f40d47e54a429a9f55ae410e70ddf3"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:55.677335 24246 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:55.681936 22576 raft_consensus.cc:482] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.682060 22576 raft_consensus.cc:3037] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.684753 22576 raft_consensus.cc:504] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.689507 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40d47e54a429a9f55ae410e70ddf3" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.689723 22706 raft_consensus.cc:3037] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.690284 22166 catalog_manager.cc:5484] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.692203 22706 raft_consensus.cc:2454] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 3.
I20240502 14:15:55.692678 22509 leader_election.cc:302] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:55.692983 22576 leader_election.cc:288] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 3 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.693331 24712 raft_consensus.cc:2781] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.693249 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f40d47e54a429a9f55ae410e70ddf3" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:55.693497 22308 raft_consensus.cc:3032] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.693576 22308 raft_consensus.cc:729] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.693933 24712 raft_consensus.cc:686] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:55.693910 22308 consensus_queue.cc:260] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.694190 22308 raft_consensus.cc:3037] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.694252 24712 consensus_queue.cc:237] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.699944 22576 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:55.701526 22308 raft_consensus.cc:2454] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 3.
I20240502 14:15:55.704217 22576 raft_consensus.cc:482] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.704373 22576 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.707479 22576 raft_consensus.cc:504] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:55.708024 24643 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:55.708572 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.708815 22445 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.711457 22445 raft_consensus.cc:2454] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.713506 22512 leader_election.cc:302] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:55.711833 22576 leader_election.cc:288] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.715168 22166 catalog_manager.cc:5484] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 2 to 3, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 3 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.715837 22571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7951641f78914cc583da7f0c4b99ed9d"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:55.715960 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.716076 22571 raft_consensus.cc:482] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.716147 22706 raft_consensus.cc:3032] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.716236 22571 raft_consensus.cc:3037] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.716257 22706 raft_consensus.cc:729] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.716482 22706 consensus_queue.cc:260] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:55.716869 22706 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.717787 24712 raft_consensus.cc:2781] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:55.717922 24712 raft_consensus.cc:686] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:55.718196 24712 consensus_queue.cc:237] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:55.719333 22706 raft_consensus.cc:2454] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.719492 22571 raft_consensus.cc:504] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.720902 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7951641f78914cc583da7f0c4b99ed9d" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.721122 22707 raft_consensus.cc:3037] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.721426 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7951641f78914cc583da7f0c4b99ed9d" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:55.721675 22308 raft_consensus.cc:3032] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:55.721762 22308 raft_consensus.cc:729] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.722052 22308 consensus_queue.cc:260] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.722378 22308 raft_consensus.cc:3037] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:55.722900 22571 leader_election.cc:288] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 3 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.723992 22707 raft_consensus.cc:2454] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 3.
I20240502 14:15:55.726156 22308 raft_consensus.cc:2396] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 00c39ff1d7004a8f8a8847aad597f242 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20240502 14:15:55.724459 22509 leader_election.cc:302] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:55.726867 24852 raft_consensus.cc:2781] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:55.727068 24852 raft_consensus.cc:686] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [term 3 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:55.727296 24852 consensus_queue.cc:237] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:55.729849 22571 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:55.730063 22571 raft_consensus.cc:482] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.730178 22571 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.731284 22445 raft_consensus.cc:1261] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.731918 24809 consensus_queue.cc:1035] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.733182 22574 raft_consensus.cc:1261] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.734037 24809 consensus_queue.cc:1035] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.735720 22571 raft_consensus.cc:504] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.736330 22442 raft_consensus.cc:1261] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 14:15:55.736565 22443 raft_consensus.cc:1261] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.737051 24809 consensus_queue.cc:1035] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.739287 24811 consensus_queue.cc:1035] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.744921 22576 raft_consensus.cc:1261] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.745682 24811 consensus_queue.cc:1035] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.748615 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.748771 22706 raft_consensus.cc:3032] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.748850 22706 raft_consensus.cc:729] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.750092 22574 raft_consensus.cc:1261] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.750583 24811 consensus_queue.cc:1035] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.750576 22706 consensus_queue.cc:260] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.751034 22706 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.751689 22442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.751853 22442 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.754482 22442 raft_consensus.cc:2454] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.754879 22512 leader_election.cc:302] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:55.755209 22444 raft_consensus.cc:1261] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.755306 24712 raft_consensus.cc:2781] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:55.758455 24810 consensus_queue.cc:1035] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.759109 22576 raft_consensus.cc:1261] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.760792 24809 consensus_queue.cc:1035] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.761337 22706 raft_consensus.cc:2454] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.764361 22571 leader_election.cc:288] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:55.765977 22575 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:55.766196 22575 raft_consensus.cc:482] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.766300 22575 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.769757 22575 raft_consensus.cc:504] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:55.771037 22706 raft_consensus.cc:1261] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 14:15:55.771123 22706 pending_rounds.cc:76] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10: Aborting all ops after (but not including) 3
I20240502 14:15:55.771198 22706 pending_rounds.cc:97] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 14:15:55.771263 22706 raft_consensus.cc:2866] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:55.771343 22575 leader_election.cc:288] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:55.771816 24809 consensus_queue.cc:1035] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.771960 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:55.772130 22707 raft_consensus.cc:3032] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.772217 22707 raft_consensus.cc:729] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:55.773001 22707 consensus_queue.cc:260] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:55.773380 22707 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.775923 22707 raft_consensus.cc:2454] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.776455 22509 leader_election.cc:302] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:55.777019 24874 raft_consensus.cc:2781] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:55.777144 24874 raft_consensus.cc:686] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:55.777524 24874 consensus_queue.cc:237] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:55.777884 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.778120 22445 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.779107 22575 raft_consensus.cc:1261] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 14:15:55.779620 22444 raft_consensus.cc:1261] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.779724 24316 consensus_queue.cc:1035] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.780620 22445 raft_consensus.cc:2454] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.782392 24810 consensus_queue.cc:1035] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.785063 22573 raft_consensus.cc:1261] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.785799 24810 consensus_queue.cc:1035] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.786309 24712 raft_consensus.cc:686] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:55.789212 24712 consensus_queue.cc:237] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:55.803571 22707 raft_consensus.cc:1261] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 14:15:55.804068 22571 raft_consensus.cc:1261] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 14:15:55.804601 24810 consensus_queue.cc:1035] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.808248 22291 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240502 14:15:55.808208 22576 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:55.808362 22576 raft_consensus.cc:482] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.808478 22576 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.820709 22708 raft_consensus.cc:1261] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 14:15:55.820834 22707 pending_rounds.cc:76] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10: Aborting all ops after (but not including) 3
I20240502 14:15:55.820927 22708 pending_rounds.cc:76] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10: Aborting all ops after (but not including) 3
I20240502 14:15:55.820981 22708 pending_rounds.cc:97] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 14:15:55.821094 22708 raft_consensus.cc:2866] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:55.821484 22707 pending_rounds.cc:97] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 14:15:55.821581 22707 raft_consensus.cc:2866] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:55.821681 24810 consensus_queue.cc:1035] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.822459 24810 consensus_queue.cc:1035] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.823398 22576 raft_consensus.cc:504] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.824553 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.824820 22445 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.839826 22572 raft_consensus.cc:1226] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Rejecting Update request from peer 2b54c913a94f4c399ec08cf222a941e6 for earlier term 3. Current term is 4. Ops: []
I20240502 14:15:55.840373 24810 consensus_queue.cc:1046] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.840651 24810 raft_consensus.cc:3032] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.840764 24810 raft_consensus.cc:729] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.841023 24810 consensus_queue.cc:260] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.841394 24810 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.844831 22572 raft_consensus.cc:1261] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.845441 24811 consensus_queue.cc:1035] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.850176 22444 raft_consensus.cc:1261] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:55.851477 24793 consensus_queue.cc:1035] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.863180 22444 raft_consensus.cc:1261] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 14:15:55.863869 24793 consensus_queue.cc:1035] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.878811 22688 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240502 14:15:55.890806 22708 raft_consensus.cc:1261] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240502 14:15:55.891475 24793 consensus_queue.cc:1035] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.922667 22445 raft_consensus.cc:2454] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.923067 22443 raft_consensus.cc:1226] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Rejecting Update request from peer 2b54c913a94f4c399ec08cf222a941e6 for earlier term 3. Current term is 4. Ops: []
I20240502 14:15:55.924719 22512 leader_election.cc:302] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:55.925282 24712 raft_consensus.cc:2781] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:55.925534 24712 raft_consensus.cc:686] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:55.925956 24712 consensus_queue.cc:237] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.929126 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:55.929615 22308 raft_consensus.cc:2396] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 00c39ff1d7004a8f8a8847aad597f242 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 14:15:55.938449 22166 catalog_manager.cc:5484] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:55.941679 22166 catalog_manager.cc:5484] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 2 to 3, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 3 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.955618 22708 raft_consensus.cc:1261] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Refusing update from remote peer 2b54c913a94f4c399ec08cf222a941e6: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 14:15:55.955804 22708 pending_rounds.cc:76] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10: Aborting all ops after (but not including) 3
I20240502 14:15:55.955935 22708 pending_rounds.cc:97] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 14:15:55.956053 22708 raft_consensus.cc:2866] T 64ec9e34ad4d49faad727155411f9762 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:55.956571 24316 consensus_queue.cc:1035] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:55.960971 22576 leader_election.cc:288] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:55.962402 22574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:55.962569 22574 raft_consensus.cc:482] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.962683 22574 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.965888 22574 raft_consensus.cc:504] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.966287 24511 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:55.967043 22574 leader_election.cc:288] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:55.967005 22443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.967263 22443 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.968322 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:55.968430 22708 raft_consensus.cc:1261] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 14:15:55.968519 22308 raft_consensus.cc:3032] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:55.968600 22308 raft_consensus.cc:729] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:55.968904 24852 consensus_queue.cc:1035] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.968875 22308 consensus_queue.cc:260] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.969213 22308 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.969825 22443 raft_consensus.cc:2454] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.971966 22512 leader_election.cc:302] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:55.972002 22308 raft_consensus.cc:2454] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.972219 24874 raft_consensus.cc:2781] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:55.972376 24874 raft_consensus.cc:686] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:55.972637 24874 consensus_queue.cc:237] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.972826 22573 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:55.972992 22573 raft_consensus.cc:482] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:55.973109 22573 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.974439 22442 raft_consensus.cc:1261] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 14:15:55.975237 24874 consensus_queue.cc:1035] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:55.976222 22573 raft_consensus.cc:504] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.977061 22442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:55.977214 22442 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:55.979745 22442 raft_consensus.cc:2454] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:55.980268 22512 leader_election.cc:302] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:55.980537 24874 raft_consensus.cc:2781] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:55.980682 24874 raft_consensus.cc:686] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:55.980917 24874 consensus_queue.cc:237] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:55.981587 22165 catalog_manager.cc:5484] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.982764 22165 catalog_manager.cc:5484] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:55.983412 22165 catalog_manager.cc:5484] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.991060 22165 catalog_manager.cc:5484] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:55.995235 22165 catalog_manager.cc:5484] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.016781 24378 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240502 14:15:56.025382 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.025646 22308 raft_consensus.cc:3032] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:56.025728 22308 raft_consensus.cc:729] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:56.026042 22308 consensus_queue.cc:260] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.026449 22308 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.029343 22308 raft_consensus.cc:2454] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.029812 22573 leader_election.cc:288] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:56.030841 22576 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:56.031049 22576 raft_consensus.cc:482] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.031154 22576 raft_consensus.cc:3037] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:56.033869 22576 raft_consensus.cc:504] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.034940 22444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:56.035189 22444 raft_consensus.cc:3037] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:56.035405 22576 leader_election.cc:288] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 3 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.035429 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.035673 22708 raft_consensus.cc:3032] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:56.035739 22708 raft_consensus.cc:729] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:56.035977 22708 consensus_queue.cc:260] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.036237 22574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:56.036281 22708 raft_consensus.cc:3037] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:56.036401 22574 raft_consensus.cc:482] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.036482 22574 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.037863 22444 raft_consensus.cc:2454] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 3.
I20240502 14:15:56.038617 22512 leader_election.cc:302] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:56.038820 22708 raft_consensus.cc:2454] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 3.
I20240502 14:15:56.038946 24873 raft_consensus.cc:2781] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:56.039001 22574 raft_consensus.cc:504] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.039691 22443 raft_consensus.cc:1261] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 14:15:56.039899 22574 leader_election.cc:288] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.039120 24873 raft_consensus.cc:686] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.040412 24873 consensus_queue.cc:237] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.040593 22443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:56.040783 22443 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.041576 24873 consensus_queue.cc:1035] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.044190 22443 raft_consensus.cc:2454] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.045194 22512 leader_election.cc:302] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:56.045502 24873 raft_consensus.cc:2781] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:56.045543 22707 raft_consensus.cc:1261] T 5f79b02ca45b4dea90513712bc57e137 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 14:15:56.045632 24873 raft_consensus.cc:686] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.045917 24873 consensus_queue.cc:237] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.045789 22165 catalog_manager.cc:5484] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 2 to 3, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 3 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.047140 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.047358 22708 raft_consensus.cc:3032] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:56.047434 22708 raft_consensus.cc:729] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:56.047741 22708 consensus_queue.cc:260] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.048144 22708 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.050830 22708 raft_consensus.cc:2454] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.051158 24712 consensus_queue.cc:1035] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.053642 22574 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:56.053831 22574 raft_consensus.cc:482] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.054003 22574 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.054863 22706 raft_consensus.cc:1261] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 14:15:56.058779 22574 raft_consensus.cc:504] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.059207 24852 consensus_queue.cc:1035] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.060199 22574 leader_election.cc:288] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:56.060750 22706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.060943 22706 raft_consensus.cc:3032] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:56.061018 22706 raft_consensus.cc:729] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:56.061280 22706 consensus_queue.cc:260] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.061594 22706 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.062011 22308 raft_consensus.cc:1261] T 5f79b02ca45b4dea90513712bc57e137 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 14:15:56.064083 22706 raft_consensus.cc:2454] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.064363 24852 consensus_queue.cc:1035] T 5f79b02ca45b4dea90513712bc57e137 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.064841 22572 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "cd6972bee4d94830ac2d6a5660adc944"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:56.065006 22572 raft_consensus.cc:482] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.065116 22572 raft_consensus.cc:3037] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.065042 22509 leader_election.cc:302] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:56.065749 24852 raft_consensus.cc:2781] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:56.065920 24852 raft_consensus.cc:686] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.066181 24852 consensus_queue.cc:237] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.067620 22572 raft_consensus.cc:504] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.068562 22572 leader_election.cc:288] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:56.068862 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6972bee4d94830ac2d6a5660adc944" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.069126 22312 raft_consensus.cc:3032] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:56.069205 22312 raft_consensus.cc:729] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:56.069521 22312 consensus_queue.cc:260] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.069933 22312 raft_consensus.cc:3037] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.071048 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd6972bee4d94830ac2d6a5660adc944" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:56.071198 22445 raft_consensus.cc:3037] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.072547 22312 raft_consensus.cc:2454] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.073146 22509 leader_election.cc:302] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:56.073565 22445 raft_consensus.cc:2454] T cd6972bee4d94830ac2d6a5660adc944 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.073691 24712 raft_consensus.cc:2781] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:56.073830 24712 raft_consensus.cc:686] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.074092 24712 consensus_queue.cc:237] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.074267 22572 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a53d97bb173340c586fe577b79ca5a55"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:56.074451 22572 raft_consensus.cc:482] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.074554 22572 raft_consensus.cc:3037] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.078159 22572 raft_consensus.cc:504] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.078964 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53d97bb173340c586fe577b79ca5a55" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.079118 22707 raft_consensus.cc:3037] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.081408 22443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:56.081566 22443 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.083972 22443 raft_consensus.cc:2454] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.084968 22707 raft_consensus.cc:2454] T a53d97bb173340c586fe577b79ca5a55 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.085336 22509 leader_election.cc:302] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:56.085628 24873 raft_consensus.cc:2781] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:56.085850 24873 raft_consensus.cc:686] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.086105 24873 consensus_queue.cc:237] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.088392 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a53d97bb173340c586fe577b79ca5a55" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.088706 22308 raft_consensus.cc:3032] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:56.088831 22308 raft_consensus.cc:729] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:56.089577 22308 consensus_queue.cc:260] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.089974 22308 raft_consensus.cc:3037] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.089921 22162 catalog_manager.cc:5484] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.092464 22308 raft_consensus.cc:2454] T a53d97bb173340c586fe577b79ca5a55 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.100248 22572 leader_election.cc:288] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:56.101544 22572 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4c8f2b775fa54c44a5d855d586576696"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:56.101774 22572 raft_consensus.cc:482] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.101971 22572 raft_consensus.cc:3037] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:56.106499 22572 raft_consensus.cc:504] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.107797 22162 catalog_manager.cc:5484] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.113183 22162 catalog_manager.cc:5484] T a53d97bb173340c586fe577b79ca5a55 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.114414 22572 leader_election.cc:288] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 3 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.114596 22162 catalog_manager.cc:5484] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.115141 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8f2b775fa54c44a5d855d586576696" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.115386 22708 raft_consensus.cc:3037] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:56.115418 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c8f2b775fa54c44a5d855d586576696" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.115613 22308 raft_consensus.cc:3032] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 LEADER]: Stepping down as leader of term 2
I20240502 14:15:56.115705 22308 raft_consensus.cc:729] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:56.115958 22308 consensus_queue.cc:260] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.116309 22308 raft_consensus.cc:3037] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 2 FOLLOWER]: Advancing to term 3
I20240502 14:15:56.118556 22708 raft_consensus.cc:2454] T 4c8f2b775fa54c44a5d855d586576696 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 3.
I20240502 14:15:56.118963 22308 raft_consensus.cc:2454] T 4c8f2b775fa54c44a5d855d586576696 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 3.
I20240502 14:15:56.119128 22509 leader_election.cc:302] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:56.120065 24872 raft_consensus.cc:2781] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Leader election won for term 3
I20240502 14:15:56.120218 24872 raft_consensus.cc:686] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.120512 24872 consensus_queue.cc:237] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.122038 22572 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:56.122233 22572 raft_consensus.cc:482] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.122349 22572 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.131875 22162 catalog_manager.cc:5484] T 4c8f2b775fa54c44a5d855d586576696 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 2 to 3, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 3 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.132668 22572 raft_consensus.cc:504] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.134138 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.134305 22308 raft_consensus.cc:3032] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:56.134402 22308 raft_consensus.cc:729] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:56.134611 22308 consensus_queue.cc:260] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.134948 22308 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.137420 22308 raft_consensus.cc:2454] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.137923 22509 leader_election.cc:302] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:56.138190 24873 raft_consensus.cc:2781] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:56.138317 24873 raft_consensus.cc:686] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.138532 24873 consensus_queue.cc:237] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.144076 22165 catalog_manager.cc:5484] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.146543 22572 leader_election.cc:288] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:56.147377 22443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:56.147609 22443 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.148399 22572 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f9da0fcab2294a3fb62c132a6b60214c"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:56.148604 22572 raft_consensus.cc:482] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.148756 22572 raft_consensus.cc:3037] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.150424 22443 raft_consensus.cc:2454] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.151396 22572 raft_consensus.cc:504] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.152052 22572 leader_election.cc:288] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:56.152695 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9da0fcab2294a3fb62c132a6b60214c" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.152675 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9da0fcab2294a3fb62c132a6b60214c" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.152873 22708 raft_consensus.cc:3037] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.152879 22308 raft_consensus.cc:3032] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:56.152968 22308 raft_consensus.cc:729] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:56.153208 22308 consensus_queue.cc:260] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.153606 22308 raft_consensus.cc:3037] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.155630 22708 raft_consensus.cc:2454] T f9da0fcab2294a3fb62c132a6b60214c P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.156037 22509 leader_election.cc:302] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:56.156365 24872 raft_consensus.cc:2781] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:56.156518 24872 raft_consensus.cc:686] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.156755 24872 consensus_queue.cc:237] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.157183 22572 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:56.157436 22572 raft_consensus.cc:482] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.157552 22572 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.158548 22443 raft_consensus.cc:1261] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 14:15:56.159119 24873 consensus_queue.cc:1035] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.160045 22572 raft_consensus.cc:504] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.160799 22572 leader_election.cc:288] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183)
I20240502 14:15:56.164233 22445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:56.164434 22445 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.164547 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.164729 22312 raft_consensus.cc:3032] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:56.164814 22312 raft_consensus.cc:729] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:56.165045 22312 consensus_queue.cc:260] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.165406 22312 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.166971 22445 raft_consensus.cc:2454] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.167366 22512 leader_election.cc:302] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:56.167692 24873 raft_consensus.cc:2781] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:56.167809 24873 raft_consensus.cc:686] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.167855 22312 raft_consensus.cc:2454] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.168063 24873 consensus_queue.cc:237] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.172464 22165 catalog_manager.cc:5484] T f9da0fcab2294a3fb62c132a6b60214c P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.173664 22308 raft_consensus.cc:2454] T f9da0fcab2294a3fb62c132a6b60214c P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.173852 22708 raft_consensus.cc:1261] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 14:15:56.174499 24873 consensus_queue.cc:1035] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.180490 22572 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c"
dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
from {username='slave'} at 127.0.0.1:38316
I20240502 14:15:56.180667 22572 raft_consensus.cc:482] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.181660 22572 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.184023 22165 catalog_manager.cc:5484] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.187951 22572 raft_consensus.cc:504] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.188490 22708 raft_consensus.cc:1261] T 44f40d47e54a429a9f55ae410e70ddf3 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:56.188941 22572 leader_election.cc:288] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Requested vote from peers 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194:39183), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:56.188992 22443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
I20240502 14:15:56.189191 22443 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.191119 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c" candidate_uuid: "00c39ff1d7004a8f8a8847aad597f242" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.191311 22308 raft_consensus.cc:3032] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:56.191409 22308 raft_consensus.cc:729] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:56.191638 22443 raft_consensus.cc:2454] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.191753 22308 consensus_queue.cc:260] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.192011 22512 leader_election.cc:302] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:56.192184 22308 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.192258 24712 consensus_queue.cc:1035] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:56.192628 24712 raft_consensus.cc:2781] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:56.192749 24712 raft_consensus.cc:686] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Leader. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.192970 24712 consensus_queue.cc:237] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.194985 22308 raft_consensus.cc:2454] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 00c39ff1d7004a8f8a8847aad597f242 in term 4.
I20240502 14:15:56.196874 22162 catalog_manager.cc:5484] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 reported cstate change: term changed from 3 to 4, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195). New cstate: current_term: 4 leader_uuid: "00c39ff1d7004a8f8a8847aad597f242" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.202589 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.202814 22443 raft_consensus.cc:482] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.202976 22443 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.205699 22443 raft_consensus.cc:504] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.210747 22308 raft_consensus.cc:1261] T 44f40d47e54a429a9f55ae410e70ddf3 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240502 14:15:56.210979 22443 leader_election.cc:288] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.211382 22572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.211463 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b08f8d408bd42eb8f1f9ca80fe176c1" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.211583 22572 raft_consensus.cc:3032] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:56.211645 22708 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.211661 22572 raft_consensus.cc:729] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.211925 22572 consensus_queue.cc:260] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.212292 22572 raft_consensus.cc:3037] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.214991 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.214998 24872 consensus_queue.cc:1035] T 44f40d47e54a429a9f55ae410e70ddf3 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:56.215194 22443 raft_consensus.cc:482] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.215287 22443 raft_consensus.cc:3037] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.215736 22708 raft_consensus.cc:2454] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.216261 22572 raft_consensus.cc:2454] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 00c39ff1d7004a8f8a8847aad597f242 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.216670 22379 leader_election.cc:302] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, 017239f33db84a7b98c8c0244d55ecc5; no voters:
I20240502 14:15:56.217023 24078 raft_consensus.cc:2781] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.217175 24078 raft_consensus.cc:686] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.217541 24078 consensus_queue.cc:237] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.221685 22443 raft_consensus.cc:504] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.222743 22312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.222985 22312 raft_consensus.cc:3037] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.228937 22165 catalog_manager.cc:5484] T 5b08f8d408bd42eb8f1f9ca80fe176c1 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 4 to 5, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 5 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.230458 24078 raft_consensus.cc:482] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:56.230654 24078 raft_consensus.cc:504] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.231479 24078 leader_election.cc:288] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.232997 22309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:56.233259 22309 raft_consensus.cc:2454] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 4.
I20240502 14:15:56.233793 22377 leader_election.cc:302] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:56.234392 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:56.234735 22708 raft_consensus.cc:2396] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 14:15:56.236706 22312 raft_consensus.cc:2454] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 4.
I20240502 14:15:56.237167 22377 leader_election.cc:302] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:56.237514 24728 raft_consensus.cc:2781] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240502 14:15:56.237583 24078 raft_consensus.cc:2781] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:56.237609 24728 raft_consensus.cc:482] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:56.237917 24728 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.238983 22443 leader_election.cc:288] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 4 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.239684 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2c0dc3c5bd24a8ebea5f15f037e7a2a" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.239902 22708 raft_consensus.cc:3032] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:56.240036 22708 raft_consensus.cc:729] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:56.240319 24728 raft_consensus.cc:504] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.240368 22708 consensus_queue.cc:260] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.240859 22708 raft_consensus.cc:3037] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.240885 24728 leader_election.cc:288] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.241935 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.242112 22443 raft_consensus.cc:482] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.242241 22443 raft_consensus.cc:3037] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.243810 22309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.244076 22309 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.244766 22443 raft_consensus.cc:504] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.245718 22307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.245915 22307 raft_consensus.cc:3032] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:56.245987 22307 raft_consensus.cc:729] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:56.246194 22307 consensus_queue.cc:260] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.246486 22307 raft_consensus.cc:3037] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.247666 22309 raft_consensus.cc:2454] T 3c962be661024d9aae9d8d613763cc49 P 2b54c913a94f4c399ec08cf222a941e6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.248256 24078 raft_consensus.cc:686] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [term 4 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.248610 24078 consensus_queue.cc:237] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.248916 22307 raft_consensus.cc:2454] T eff83cace8554f40beaffe89cb1b684d P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 4.
I20240502 14:15:56.249308 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.249467 22707 raft_consensus.cc:3037] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.251984 22707 raft_consensus.cc:2396] T 3c962be661024d9aae9d8d613763cc49 P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 14:15:56.252439 22708 raft_consensus.cc:2454] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 4.
I20240502 14:15:56.252831 22377 leader_election.cc:302] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:56.253099 22377 leader_election.cc:302] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:56.254802 24728 raft_consensus.cc:2781] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.254374 22307 raft_consensus.cc:1261] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 4. (term mismatch)
I20240502 14:15:56.255262 22307 pending_rounds.cc:76] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6: Aborting all ops after (but not including) 3
I20240502 14:15:56.255380 22307 pending_rounds.cc:97] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6: Aborting uncommitted NO_OP operation due to leader change: 2.4
I20240502 14:15:56.255589 22307 raft_consensus.cc:2866] T 7951641f78914cc583da7f0c4b99ed9d P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:56.256131 24712 consensus_queue.cc:1035] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:56.256467 24078 raft_consensus.cc:2781] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:56.256639 24078 raft_consensus.cc:686] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [term 4 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.256954 24078 consensus_queue.cc:237] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.260239 22162 catalog_manager.cc:5484] T a2c0dc3c5bd24a8ebea5f15f037e7a2a P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 3 to 4, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 4 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.261440 24728 raft_consensus.cc:686] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.261687 24728 consensus_queue.cc:237] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.264717 22708 raft_consensus.cc:1261] T 7951641f78914cc583da7f0c4b99ed9d P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20240502 14:15:56.265964 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eff83cace8554f40beaffe89cb1b684d" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.266129 22707 raft_consensus.cc:3037] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.266320 24872 consensus_queue.cc:1035] T 7951641f78914cc583da7f0c4b99ed9d P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240502 14:15:56.268682 22707 raft_consensus.cc:2454] T eff83cace8554f40beaffe89cb1b684d P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 4.
I20240502 14:15:56.271385 22708 raft_consensus.cc:1261] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 14:15:56.271875 24872 consensus_queue.cc:1035] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.272675 22162 catalog_manager.cc:5484] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 3 to 4, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 4 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.275094 22445 raft_consensus.cc:1261] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 14:15:56.276520 24873 consensus_queue.cc:1035] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.277333 22162 catalog_manager.cc:5484] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 3 to 5, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 5 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.278388 22443 leader_election.cc:288] T eff83cace8554f40beaffe89cb1b684d P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 4 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.284540 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.284730 22443 raft_consensus.cc:482] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.284837 22443 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.288108 22443 raft_consensus.cc:504] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:56.288723 22443 leader_election.cc:288] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:56.289641 22572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.289855 22572 raft_consensus.cc:3032] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:56.289935 22572 raft_consensus.cc:729] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.290184 22572 consensus_queue.cc:260] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:56.290510 22572 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.290908 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc76aec8299347378f28f9ac8a8b5ce8" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.291083 22707 raft_consensus.cc:3037] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.294286 22707 raft_consensus.cc:2454] T fc76aec8299347378f28f9ac8a8b5ce8 P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.294657 22377 leader_election.cc:302] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:56.294945 24897 raft_consensus.cc:2781] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.295064 24897 raft_consensus.cc:686] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.295315 24897 consensus_queue.cc:237] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:56.296257 22445 raft_consensus.cc:1261] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 14:15:56.296783 24712 consensus_queue.cc:1035] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.298007 22572 raft_consensus.cc:2454] T fc76aec8299347378f28f9ac8a8b5ce8 P 00c39ff1d7004a8f8a8847aad597f242 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.298707 22166 catalog_manager.cc:5484] T fc76aec8299347378f28f9ac8a8b5ce8 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 4 to 5, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 5 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.300024 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.300173 22443 raft_consensus.cc:482] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.300288 22443 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.302666 22443 raft_consensus.cc:504] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.304219 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.304525 22308 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.305742 22707 raft_consensus.cc:1261] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 14:15:56.306074 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa2f5692b87f4ae0983006b09ffc4cbd" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.306241 22576 raft_consensus.cc:3032] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:56.306351 24712 consensus_queue.cc:1035] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.306350 22576 raft_consensus.cc:729] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.306914 22576 consensus_queue.cc:260] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.307197 22576 raft_consensus.cc:3037] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.307718 22308 raft_consensus.cc:2454] T fa2f5692b87f4ae0983006b09ffc4cbd P 2b54c913a94f4c399ec08cf222a941e6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.308151 22377 leader_election.cc:302] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:56.308507 24897 raft_consensus.cc:2781] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.308632 24897 raft_consensus.cc:686] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.308820 22443 leader_election.cc:288] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:56.308853 24897 consensus_queue.cc:237] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.309834 22576 raft_consensus.cc:2396] T fa2f5692b87f4ae0983006b09ffc4cbd P 00c39ff1d7004a8f8a8847aad597f242 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 14:15:56.310453 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.310948 22443 raft_consensus.cc:482] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.311053 22443 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.313623 22166 catalog_manager.cc:5484] T fa2f5692b87f4ae0983006b09ffc4cbd P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 4 to 5, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 5 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:56.318343 22443 raft_consensus.cc:504] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.319480 22308 raft_consensus.cc:1261] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20240502 14:15:56.319669 22308 pending_rounds.cc:76] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6: Aborting all ops after (but not including) 4
I20240502 14:15:56.319803 22308 pending_rounds.cc:97] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20240502 14:15:56.319991 22308 raft_consensus.cc:2866] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20240502 14:15:56.320940 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.321134 22576 raft_consensus.cc:3032] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:56.321050 24712 consensus_queue.cc:1035] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.321252 22576 raft_consensus.cc:729] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.321470 22576 consensus_queue.cc:260] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.321728 22576 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.324106 22576 raft_consensus.cc:2396] T eb34b5ac0e8e4c15a1df9a62e439c56c P 00c39ff1d7004a8f8a8847aad597f242 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 14:15:56.324862 24896 raft_consensus.cc:482] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:56.325191 22445 raft_consensus.cc:1261] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20240502 14:15:56.325958 24852 consensus_queue.cc:1035] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240502 14:15:56.325052 24896 raft_consensus.cc:504] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.327194 24896 leader_election.cc:288] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.327898 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10" is_pre_election: true
I20240502 14:15:56.327991 22307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:56.328212 22707 raft_consensus.cc:2396] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 14:15:56.328197 22307 raft_consensus.cc:2454] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 4.
I20240502 14:15:56.328629 22377 leader_election.cc:302] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters: c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:56.328940 24897 raft_consensus.cc:2781] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240502 14:15:56.329048 24897 raft_consensus.cc:482] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:56.329146 24897 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.329958 22443 leader_election.cc:288] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:56.330039 22307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb34b5ac0e8e4c15a1df9a62e439c56c" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.330232 22307 raft_consensus.cc:3037] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.331243 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3c962be661024d9aae9d8d613763cc49"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.331405 22443 raft_consensus.cc:465] T 3c962be661024d9aae9d8d613763cc49 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Not starting forced leader election -- already a leader
I20240502 14:15:56.332347 22307 raft_consensus.cc:2454] T eb34b5ac0e8e4c15a1df9a62e439c56c P 2b54c913a94f4c399ec08cf222a941e6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.332373 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.332513 22443 raft_consensus.cc:482] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.332629 22443 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.332717 22377 leader_election.cc:302] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters: 00c39ff1d7004a8f8a8847aad597f242
I20240502 14:15:56.333050 24896 raft_consensus.cc:2781] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.333186 24896 raft_consensus.cc:686] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.333463 24896 consensus_queue.cc:237] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.335361 22443 raft_consensus.cc:504] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.336184 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.336231 22443 leader_election.cc:288] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:56.336375 22707 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.336658 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a15c7effaee47b992ffd5078bf7ca9d" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.336830 22576 raft_consensus.cc:3032] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:56.336902 22576 raft_consensus.cc:729] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.337107 22576 consensus_queue.cc:260] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.337456 22576 raft_consensus.cc:3037] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.337188 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.337714 22443 raft_consensus.cc:482] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.337879 22443 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.338864 22707 raft_consensus.cc:2454] T 4a15c7effaee47b992ffd5078bf7ca9d P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.339351 22377 leader_election.cc:302] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:56.339599 24728 raft_consensus.cc:2781] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.339720 24728 raft_consensus.cc:686] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.340049 22576 raft_consensus.cc:2454] T 4a15c7effaee47b992ffd5078bf7ca9d P 00c39ff1d7004a8f8a8847aad597f242 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.339974 24728 consensus_queue.cc:237] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.340938 22443 raft_consensus.cc:504] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.341737 24897 raft_consensus.cc:504] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.342052 22307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.342243 22307 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.342873 24897 leader_election.cc:288] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.346362 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.346562 22707 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.349675 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812638aac21c48aab6d9dd72e08d98d4" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.345562 22307 raft_consensus.cc:2454] T 65879e88cc294682a159d1370a11c3f7 P 2b54c913a94f4c399ec08cf222a941e6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.350278 22443 leader_election.cc:288] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.350615 22308 raft_consensus.cc:3037] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.351696 22377 leader_election.cc:302] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:56.352596 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65879e88cc294682a159d1370a11c3f7" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.352653 22308 raft_consensus.cc:2454] T 812638aac21c48aab6d9dd72e08d98d4 P 2b54c913a94f4c399ec08cf222a941e6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.352774 22708 raft_consensus.cc:3037] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.353633 22377 leader_election.cc:302] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:56.353888 24896 raft_consensus.cc:2781] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.354019 24896 raft_consensus.cc:686] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.354084 24728 raft_consensus.cc:2781] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.354255 24728 raft_consensus.cc:686] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.354652 22707 raft_consensus.cc:2396] T 812638aac21c48aab6d9dd72e08d98d4 P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 14:15:56.354521 24728 consensus_queue.cc:237] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.354811 22166 catalog_manager.cc:5484] T 4a15c7effaee47b992ffd5078bf7ca9d P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 4 to 5, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 5 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:56.355322 22708 raft_consensus.cc:2396] T 65879e88cc294682a159d1370a11c3f7 P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 14:15:56.355463 24896 consensus_queue.cc:237] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.355365 22166 catalog_manager.cc:5484] T eb34b5ac0e8e4c15a1df9a62e439c56c P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 4 to 5, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 5 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.358258 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.358417 22443 raft_consensus.cc:482] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.359654 22443 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.362928 22443 raft_consensus.cc:504] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.363952 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.364252 22708 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.364305 22443 leader_election.cc:288] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:56.364684 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f84542c71674d618f57fdbf4e5165d5" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.364930 22308 raft_consensus.cc:3037] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.368178 22708 raft_consensus.cc:2396] T 4f84542c71674d618f57fdbf4e5165d5 P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 14:15:56.368948 22445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.369078 22308 raft_consensus.cc:2454] T 4f84542c71674d618f57fdbf4e5165d5 P 2b54c913a94f4c399ec08cf222a941e6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.369503 22377 leader_election.cc:302] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters: c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:56.370357 24896 raft_consensus.cc:2781] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.370707 24896 raft_consensus.cc:686] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.371043 24896 consensus_queue.cc:237] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.371222 22445 raft_consensus.cc:482] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.371354 22445 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.372440 22166 catalog_manager.cc:5484] T 65879e88cc294682a159d1370a11c3f7 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 3 to 5, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 5 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.372953 22166 catalog_manager.cc:5484] T 812638aac21c48aab6d9dd72e08d98d4 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 3 to 5, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 5 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:56.373878 22445 raft_consensus.cc:504] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:56.374455 22445 leader_election.cc:288] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:56.374791 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.374960 22708 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.375563 22445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.375730 22445 raft_consensus.cc:482] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.375840 22445 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.375767 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0af05ecb918d4f95ae1847dbe62e73fc" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.376071 22576 raft_consensus.cc:3032] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:56.376197 22576 raft_consensus.cc:729] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.376454 22576 consensus_queue.cc:260] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:56.376838 22576 raft_consensus.cc:3037] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.378132 22708 raft_consensus.cc:2454] T 0af05ecb918d4f95ae1847dbe62e73fc P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.378532 22377 leader_election.cc:302] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:56.378788 24896 raft_consensus.cc:2781] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.379006 24896 raft_consensus.cc:686] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.379283 24896 consensus_queue.cc:237] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:56.379904 22576 raft_consensus.cc:2454] T 0af05ecb918d4f95ae1847dbe62e73fc P 00c39ff1d7004a8f8a8847aad597f242 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.382344 22445 raft_consensus.cc:504] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.382792 22445 leader_election.cc:288] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:56.383190 22166 catalog_manager.cc:5484] T 4f84542c71674d618f57fdbf4e5165d5 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 3 to 5, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 5 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.383492 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.383653 22308 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.383687 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84ccfc433f3f4cc4bbfcba08574f6c3f" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.383622 22166 catalog_manager.cc:5484] T 0af05ecb918d4f95ae1847dbe62e73fc P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 4 to 5, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 5 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.383970 22576 raft_consensus.cc:3032] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:56.384084 22576 raft_consensus.cc:729] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.384335 22576 consensus_queue.cc:260] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.384680 22576 raft_consensus.cc:3037] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.386197 22308 raft_consensus.cc:2454] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 2b54c913a94f4c399ec08cf222a941e6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.387069 22377 leader_election.cc:302] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:56.387317 22445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.387478 22445 raft_consensus.cc:482] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.387573 22445 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.387959 24896 raft_consensus.cc:2781] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.388120 24896 raft_consensus.cc:686] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.388443 22576 raft_consensus.cc:2396] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 00c39ff1d7004a8f8a8847aad597f242 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 14:15:56.388345 24896 consensus_queue.cc:237] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.390726 22445 raft_consensus.cc:504] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:56.391175 22445 leader_election.cc:288] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:56.391103 22166 catalog_manager.cc:5484] T 84ccfc433f3f4cc4bbfcba08574f6c3f P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 4 to 5, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 5 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.391989 22445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.392115 22445 raft_consensus.cc:482] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.392201 22445 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.394539 22445 raft_consensus.cc:504] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.395066 22445 leader_election.cc:288] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:56.395902 22445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.396045 22445 raft_consensus.cc:482] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.396121 22445 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.398315 22445 raft_consensus.cc:504] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.398758 22308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.398922 22308 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.398877 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ea274a5cbef491b9803ace227431a54" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.399055 22707 raft_consensus.cc:3037] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.399415 22307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.399586 22307 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.401531 22308 raft_consensus.cc:2454] T 5ea274a5cbef491b9803ace227431a54 P 2b54c913a94f4c399ec08cf222a941e6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.401608 22707 raft_consensus.cc:2396] T 5ea274a5cbef491b9803ace227431a54 P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 14:15:56.402006 22377 leader_election.cc:302] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters: c8a8e31da9734e118e5095919ae17f10
I20240502 14:15:56.402223 22307 raft_consensus.cc:2454] T 4ef6bc006a79451babe394ba112dd717 P 2b54c913a94f4c399ec08cf222a941e6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.402365 24896 raft_consensus.cc:2781] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.402499 24896 raft_consensus.cc:686] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.402593 22377 leader_election.cc:302] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:56.402762 24896 consensus_queue.cc:237] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.403142 24728 raft_consensus.cc:2781] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.403394 24728 raft_consensus.cc:686] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.403357 22572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef6bc006a79451babe394ba112dd717" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.403546 22572 raft_consensus.cc:3032] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:56.403626 22572 raft_consensus.cc:729] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.403777 24728 consensus_queue.cc:237] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.403868 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.403873 22572 consensus_queue.cc:260] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.404050 22708 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.404134 22572 raft_consensus.cc:3037] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.406711 22572 raft_consensus.cc:2396] T 4ef6bc006a79451babe394ba112dd717 P 00c39ff1d7004a8f8a8847aad597f242 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 14:15:56.407370 22445 leader_election.cc:288] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:56.408311 22445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.408457 22445 raft_consensus.cc:482] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.408568 22445 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.408617 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a5c477465164ee2aa6d679aff875079" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.408834 22576 raft_consensus.cc:3032] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:56.408952 22576 raft_consensus.cc:729] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.408946 22708 raft_consensus.cc:2454] T 2a5c477465164ee2aa6d679aff875079 P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.409281 22377 leader_election.cc:302] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:56.409513 24728 raft_consensus.cc:2781] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.409744 24728 raft_consensus.cc:686] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.409999 24728 consensus_queue.cc:237] T 2a5c477465164ee2aa6d679aff875079 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:56.411180 24467 raft_consensus.cc:482] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240502 14:15:56.411338 24467 raft_consensus.cc:504] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.409215 22576 consensus_queue.cc:260] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } }
I20240502 14:15:56.411823 22576 raft_consensus.cc:3037] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.412067 24467 leader_election.cc:288] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:56.412653 22445 raft_consensus.cc:504] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.413177 22307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6" is_pre_election: true
I20240502 14:15:56.413555 22307 raft_consensus.cc:2396] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 14:15:56.413514 22573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.413687 22573 raft_consensus.cc:3032] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:56.413764 22573 raft_consensus.cc:729] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.413990 22307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33901f8df4ce435ca5c953d31eddcb28" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.414000 22573 consensus_queue.cc:260] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.414136 22307 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.414238 22573 raft_consensus.cc:3037] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.414798 22572 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242" is_pre_election: true
I20240502 14:15:56.415108 22572 raft_consensus.cc:2454] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 4.
I20240502 14:15:56.415661 22642 leader_election.cc:302] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters: 2b54c913a94f4c399ec08cf222a941e6
I20240502 14:15:56.416550 24467 raft_consensus.cc:2781] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20240502 14:15:56.416683 24467 raft_consensus.cc:482] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240502 14:15:56.416771 24467 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.416790 22307 raft_consensus.cc:2454] T 33901f8df4ce435ca5c953d31eddcb28 P 2b54c913a94f4c399ec08cf222a941e6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.416913 22573 raft_consensus.cc:2396] T 33901f8df4ce435ca5c953d31eddcb28 P 00c39ff1d7004a8f8a8847aad597f242 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20240502 14:15:56.417187 22377 leader_election.cc:302] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:56.417492 24728 raft_consensus.cc:2781] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.417671 24728 raft_consensus.cc:686] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.417892 24728 consensus_queue.cc:237] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.418720 22445 leader_election.cc:288] T 33901f8df4ce435ca5c953d31eddcb28 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:56.419579 22576 raft_consensus.cc:2454] T 2a5c477465164ee2aa6d679aff875079 P 00c39ff1d7004a8f8a8847aad597f242 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.419600 22445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.419749 22445 raft_consensus.cc:482] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.419847 22445 raft_consensus.cc:3037] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.421027 24467 raft_consensus.cc:504] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.422896 22307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.423151 22307 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.423579 22445 raft_consensus.cc:504] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.424382 22573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.424585 22573 raft_consensus.cc:3032] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:56.424669 22573 raft_consensus.cc:729] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.424727 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e4e12e267d2482d83dfe8c14f942d25" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.424890 22708 raft_consensus.cc:3037] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.424917 22573 consensus_queue.cc:260] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.425172 22573 raft_consensus.cc:3037] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.425455 24467 leader_election.cc:288] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 5 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:56.426962 22307 raft_consensus.cc:2396] T b9e35e0f95454e54a3603a7008026f0f P 2b54c913a94f4c399ec08cf222a941e6 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate c8a8e31da9734e118e5095919ae17f10 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 14:15:56.427384 22708 raft_consensus.cc:2454] T 1e4e12e267d2482d83dfe8c14f942d25 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 4.
I20240502 14:15:56.427592 22573 raft_consensus.cc:2396] T 1e4e12e267d2482d83dfe8c14f942d25 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20240502 14:15:56.427793 22377 leader_election.cc:302] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:56.428041 24728 raft_consensus.cc:2781] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Leader election won for term 4
I20240502 14:15:56.428184 24728 raft_consensus.cc:686] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.428143 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e35e0f95454e54a3603a7008026f0f" candidate_uuid: "c8a8e31da9734e118e5095919ae17f10" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.428308 22576 raft_consensus.cc:3037] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.428424 24728 consensus_queue.cc:237] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.430969 22576 raft_consensus.cc:2454] T b9e35e0f95454e54a3603a7008026f0f P 00c39ff1d7004a8f8a8847aad597f242 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c8a8e31da9734e118e5095919ae17f10 in term 5.
I20240502 14:15:56.431401 22642 leader_election.cc:302] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 00c39ff1d7004a8f8a8847aad597f242, c8a8e31da9734e118e5095919ae17f10; no voters: 2b54c913a94f4c399ec08cf222a941e6
I20240502 14:15:56.431869 24467 raft_consensus.cc:2781] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.432041 24467 raft_consensus.cc:686] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [term 5 LEADER]: Becoming Leader. State: Replica: c8a8e31da9734e118e5095919ae17f10, State: Running, Role: LEADER
I20240502 14:15:56.432363 24467 consensus_queue.cc:237] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.433990 22445 leader_election.cc:288] T 1e4e12e267d2482d83dfe8c14f942d25 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 4 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.435096 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.435514 22443 raft_consensus.cc:482] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.435611 22443 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.438423 22443 raft_consensus.cc:504] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.439409 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.439575 22576 raft_consensus.cc:3032] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:56.439646 22576 raft_consensus.cc:729] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.439885 22576 consensus_queue.cc:260] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.440079 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e213ccb9cf4cb7a4a3526e5af74777" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.440178 22576 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.440259 22708 raft_consensus.cc:3037] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.441145 22443 leader_election.cc:288] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.442637 22576 raft_consensus.cc:2396] T 34e213ccb9cf4cb7a4a3526e5af74777 P 00c39ff1d7004a8f8a8847aad597f242 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 017239f33db84a7b98c8c0244d55ecc5 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
I20240502 14:15:56.443135 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.443292 22443 raft_consensus.cc:482] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.443418 22443 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.443656 22166 catalog_manager.cc:5484] T 5ea274a5cbef491b9803ace227431a54 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 3 to 5, leader changed from c8a8e31da9734e118e5095919ae17f10 (127.6.196.196) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 5 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.444095 22166 catalog_manager.cc:5484] T 4ef6bc006a79451babe394ba112dd717 P 017239f33db84a7b98c8c0244d55ecc5 reported cstate change: term changed from 4 to 5, leader changed from 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195) to 017239f33db84a7b98c8c0244d55ecc5 (127.6.196.194). New cstate: current_term: 5 leader_uuid: "017239f33db84a7b98c8c0244d55ecc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } health_report { overall_health: HEALTHY } } }
I20240502 14:15:56.445020 22165 catalog_manager.cc:5484] T b9e35e0f95454e54a3603a7008026f0f P c8a8e31da9734e118e5095919ae17f10 reported cstate change: term changed from 3 to 5, leader changed from 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193) to c8a8e31da9734e118e5095919ae17f10 (127.6.196.196). New cstate: current_term: 5 leader_uuid: "c8a8e31da9734e118e5095919ae17f10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } health_report { overall_health: UNKNOWN } } }
I20240502 14:15:56.445833 22443 raft_consensus.cc:504] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.446787 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.447103 22576 raft_consensus.cc:3032] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:56.447209 22576 raft_consensus.cc:729] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.447324 22307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f344dcd458ee4654978c625337ff0bab" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.447528 22307 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.447569 22576 consensus_queue.cc:260] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.448181 22576 raft_consensus.cc:3037] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.450032 22307 raft_consensus.cc:2454] T f344dcd458ee4654978c625337ff0bab P 2b54c913a94f4c399ec08cf222a941e6 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.450556 22377 leader_election.cc:302] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, 2b54c913a94f4c399ec08cf222a941e6; no voters:
I20240502 14:15:56.451161 22576 raft_consensus.cc:2454] T f344dcd458ee4654978c625337ff0bab P 00c39ff1d7004a8f8a8847aad597f242 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.452605 22708 raft_consensus.cc:2454] T 34e213ccb9cf4cb7a4a3526e5af74777 P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.453138 22377 leader_election.cc:302] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters: 00c39ff1d7004a8f8a8847aad597f242
I20240502 14:15:56.453366 24896 raft_consensus.cc:2781] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.453366 22443 leader_election.cc:288] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203), 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463)
I20240502 14:15:56.454095 24897 raft_consensus.cc:2781] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Leader election won for term 5
I20240502 14:15:56.454282 24897 raft_consensus.cc:686] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.454607 24897 consensus_queue.cc:237] T 34e213ccb9cf4cb7a4a3526e5af74777 P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } }
I20240502 14:15:56.456259 24896 raft_consensus.cc:686] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [term 5 LEADER]: Becoming Leader. State: Replica: 017239f33db84a7b98c8c0244d55ecc5, State: Running, Role: LEADER
I20240502 14:15:56.456692 24896 consensus_queue.cc:237] T f344dcd458ee4654978c625337ff0bab P 017239f33db84a7b98c8c0244d55ecc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } } peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } }
I20240502 14:15:56.457864 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.458155 22443 raft_consensus.cc:482] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.458338 22443 raft_consensus.cc:3037] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.461831 22307 raft_consensus.cc:1261] T cd6972bee4d94830ac2d6a5660adc944 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Refusing update from remote peer 00c39ff1d7004a8f8a8847aad597f242: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20240502 14:15:56.462126 22443 raft_consensus.cc:504] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.462550 24852 consensus_queue.cc:1035] T cd6972bee4d94830ac2d6a5660adc944 P 00c39ff1d7004a8f8a8847aad597f242 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240502 14:15:56.462939 22443 leader_election.cc:288] T 64ec9e34ad4d49faad727155411f9762 P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 4 election: Requested vote from peers 2b54c913a94f4c399ec08cf222a941e6 (127.6.196.193:41463), c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959)
I20240502 14:15:56.464756 22443 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e"
dest_uuid: "017239f33db84a7b98c8c0244d55ecc5"
from {username='slave'} at 127.0.0.1:60090
I20240502 14:15:56.464941 22443 raft_consensus.cc:482] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20240502 14:15:56.465121 22443 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.465150 22307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "2b54c913a94f4c399ec08cf222a941e6"
I20240502 14:15:56.465404 22307 raft_consensus.cc:3032] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Stepping down as leader of term 3
I20240502 14:15:56.465536 22307 raft_consensus.cc:729] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 2b54c913a94f4c399ec08cf222a941e6, State: Running, Role: LEADER
I20240502 14:15:56.465817 22307 consensus_queue.cc:260] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b54c913a94f4c399ec08cf222a941e6" member_type: VOTER last_known_addr { host: "127.6.196.193" port: 41463 } } peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.466301 22307 raft_consensus.cc:3037] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 3 FOLLOWER]: Advancing to term 4
I20240502 14:15:56.467686 22443 raft_consensus.cc:504] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.468580 22443 leader_election.cc:288] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Requested vote from peers c8a8e31da9734e118e5095919ae17f10 (127.6.196.196:37959), 00c39ff1d7004a8f8a8847aad597f242 (127.6.196.195:42203)
I20240502 14:15:56.468772 22576 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "00c39ff1d7004a8f8a8847aad597f242"
I20240502 14:15:56.468966 22576 raft_consensus.cc:3032] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Stepping down as leader of term 4
I20240502 14:15:56.469041 22576 raft_consensus.cc:729] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 00c39ff1d7004a8f8a8847aad597f242, State: Running, Role: LEADER
I20240502 14:15:56.469060 22707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bf0ec2ecc7d455db867747073aaea0e" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"
I20240502 14:15:56.469285 22576 consensus_queue.cc:260] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c8a8e31da9734e118e5095919ae17f10" member_type: VOTER last_known_addr { host: "127.6.196.196" port: 37959 } } peers { permanent_uuid: "00c39ff1d7004a8f8a8847aad597f242" member_type: VOTER last_known_addr { host: "127.6.196.195" port: 42203 } } peers { permanent_uuid: "017239f33db84a7b98c8c0244d55ecc5" member_type: VOTER last_known_addr { host: "127.6.196.194" port: 39183 } }
I20240502 14:15:56.469480 22707 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.469513 22576 raft_consensus.cc:3037] T 5bf0ec2ecc7d455db867747073aaea0e P 00c39ff1d7004a8f8a8847aad597f242 [term 4 FOLLOWER]: Advancing to term 5
I20240502 14:15:56.471975 22707 raft_consensus.cc:2454] T 5bf0ec2ecc7d455db867747073aaea0e P c8a8e31da9734e118e5095919ae17f10 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 5.
I20240502 14:15:56.472229 22307 raft_consensus.cc:2454] T 64ec9e34ad4d49faad727155411f9762 P 2b54c913a94f4c399ec08cf222a941e6 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 017239f33db84a7b98c8c0244d55ecc5 in term 4.
I20240502 14:15:56.472396 22377 leader_election.cc:302] T 5bf0ec2ecc7d455db867747073aaea0e P 017239f33db84a7b98c8c0244d55ecc5 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 017239f33db84a7b98c8c0244d55ecc5, c8a8e31da9734e118e5095919ae17f10; no voters:
I20240502 14:15:56.472640 22708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64ec9e34ad4d49faad727155411f9762" candidate_uuid: "017239f33db84a7b98c8c0244d55ecc5" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: true dest_uuid: "c8a8e31da9734e118e5095919ae17f10"